{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3.12 权重衰减\n",
    "\n",
    "上一节中我们观察了过拟合现象，即模型的训练误差远小于它在测试集上的误差。虽然增大训练数据集可能会减轻过拟合，但是获取额外的训练数据往往代价高昂。本节介绍应对过拟合问题的常用方法：权重衰减（weight decay）。\n",
    "\n",
    "## 3.12.1 方法\n",
    "\n",
    "权重衰减等价于 $L_2$ 范数正则化（regularization）。正则化通过为模型损失函数添加惩罚项使学出的模型参数值较小，是应对过拟合的常用手段。我们先描述$L_2$范数正则化，再解释它为何又称权重衰减。\n",
    "\n",
    "$L_2$范数正则化在模型原损失函数基础上添加$L_2$范数惩罚项，从而得到训练所需要最小化的函数。$L_2$范数惩罚项指的是模型权重参数每个元素的平方和与一个正的常数的乘积。以3.1节（线性回归）中的线性回归损失函数\n",
    "\n",
    "$$\n",
    "\\ell(w_1, w_2, b) = \\frac{1}{n} \\sum_{i=1}^n \\frac{1}{2}\\left(x_1^{(i)} w_1 + x_2^{(i)} w_2 + b - y^{(i)}\\right)^2\n",
    "$$\n",
    "\n",
    "为例，其中$w_1, w_2$是权重参数，$b$是偏差参数，样本$i$的输入为$x_1^{(i)}, x_2^{(i)}$，标签为$y^{(i)}$，样本数为$n$。将权重参数用向量$\\boldsymbol{w} = [w_1, w_2]$表示，带有$L_2$范数惩罚项的新损失函数为\n",
    "\n",
    "$$\\ell(w_1, w_2, b) + \\frac{\\lambda}{2n} \\|\\boldsymbol{w}\\|^2,$$\n",
    "\n",
    "其中超参数$\\lambda > 0$。当权重参数均为0时，惩罚项最小。当$\\lambda$较大时，惩罚项在损失函数中的比重较大，这通常会使学到的权重参数的元素较接近0。当$\\lambda$设为0时，惩罚项完全不起作用。上式中$L_2$范数平方$\\|\\boldsymbol{w}\\|^2$展开后得到$w_1^2 + w_2^2$。有了$L_2$范数惩罚项后，在小批量随机梯度下降中，我们将线性回归一节中权重$w_1$和$w_2$的迭代方式更改为\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "w_1 &\\leftarrow \\left(1- \\frac{\\eta\\lambda}{|\\mathcal{B}|} \\right)w_1 -   \\frac{\\eta}{|\\mathcal{B}|} \\sum_{i \\in \\mathcal{B}}x_1^{(i)} \\left(x_1^{(i)} w_1 + x_2^{(i)} w_2 + b - y^{(i)}\\right),\\\\\n",
    "w_2 &\\leftarrow \\left(1- \\frac{\\eta\\lambda}{|\\mathcal{B}|} \\right)w_2 -   \\frac{\\eta}{|\\mathcal{B}|} \\sum_{i \\in \\mathcal{B}}x_2^{(i)} \\left(x_1^{(i)} w_1 + x_2^{(i)} w_2 + b - y^{(i)}\\right).\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "可见，$L_2$范数正则化令权重$w_1$和$w_2$先自乘小于1的数，再减去不含惩罚项的梯度。因此，$L_2$范数正则化又叫权重衰减。权重衰减通过惩罚绝对值较大的模型参数为需要学习的模型增加了限制，这可能对过拟合有效。实际场景中，我们有时也在惩罚项中添加偏差元素的平方和。\n",
    "\n",
    "## 3.12.2 高维线性回归实验\n",
    "\n",
    "下面，我们以高维线性回归为例来引入一个过拟合问题，并使用权重衰减来应对过拟合。设数据样本特征的维度为$p$。对于训练数据集和测试数据集中特征为$x_1, x_2, \\ldots, x_p$的任一样本，我们使用如下的线性函数来生成该样本的标签：\n",
    "\n",
    "$$\n",
    "y = 0.05 + \\sum_{i = 1}^p 0.01x_i +  \\epsilon\n",
    "$$\n",
    "\n",
    "其中噪声项$\\epsilon$服从均值为0、标准差为0.01的正态分布。为了较容易地观察过拟合，我们考虑高维线性回归问题，如设维度$p=200$；同时，我们特意把训练数据集的样本数设低，如20。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-12T02:43:19.629962Z",
     "start_time": "2020-04-12T02:43:19.619989Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.1.0\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras import layers, models, initializers, optimizers, regularizers\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-12T03:13:34.637359Z",
     "start_time": "2020-04-12T03:13:34.585942Z"
    }
   },
   "outputs": [],
   "source": [
    "n_train, n_test, num_inputs = 20, 100, 200\n",
    "true_w, true_b = tf.ones((num_inputs, 1)) * 0.01, 0.05\n",
    "\n",
    "features = tf.random.normal(shape=(n_train+n_test, num_inputs))\n",
    "labels = tf.matmul(features, true_w) + true_b\n",
    "labels += tf.random.normal(mean=0.01, shape=labels.shape)\n",
    "train_features, test_features = features[:n_train, :], features[n_train:, :]\n",
    "train_labels, test_labels = labels[: n_train], labels[n_train:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.12.3 从零开始实现\n",
    "\n",
    "下面先介绍从零开始实现权重衰减的方法。我们通过在目标函数后添加$L_2$范数惩罚项来实现权重衰减。\n",
    "\n",
    "### 3.12.3.1 初始化模型参数\n",
    "\n",
    "首先，定义随机初始化模型参数的函数。该函数为每个参数都附上梯度。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-12T03:53:03.049430Z",
     "start_time": "2020-04-12T03:53:03.045441Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function random_normal in module tensorflow.python.ops.random_ops:\n",
      "\n",
      "random_normal(shape, mean=0.0, stddev=1.0, dtype=tf.float32, seed=None, name=None)\n",
      "    Outputs random values from a normal distribution.\n",
      "    \n",
      "    Args:\n",
      "      shape: A 1-D integer Tensor or Python array. The shape of the output tensor.\n",
      "      mean: A Tensor or Python value of type `dtype`, broadcastable with `stddev`.\n",
      "        The mean of the normal distribution.\n",
      "      stddev: A Tensor or Python value of type `dtype`, broadcastable with `mean`.\n",
      "        The standard deviation of the normal distribution.\n",
      "      dtype: The type of the output.\n",
      "      seed: A Python integer. Used to create a random seed for the distribution.\n",
      "        See\n",
      "        `tf.compat.v1.set_random_seed`\n",
      "        for behavior.\n",
      "      name: A name for the operation (optional).\n",
      "    \n",
      "    Returns:\n",
      "      A tensor of the specified shape filled with random normal values.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(tf.random.normal)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 197,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-12T05:03:59.361199Z",
     "start_time": "2020-04-12T05:03:59.357209Z"
    }
   },
   "outputs": [],
   "source": [
    "def init_params():\n",
    "    w = tf.Variable(tf.random.normal(shape=(num_inputs, 1)))\n",
    "    b = tf.Variable(tf.zeros(shape=(1,)))\n",
    "    return [w, b]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.12.3.2 定义$L_2$范数惩罚项\n",
    "\n",
    "下面定义$L_2$范数惩罚项。这里只惩罚模型的权重参数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-12T05:04:01.529427Z",
     "start_time": "2020-04-12T05:04:01.526436Z"
    }
   },
   "outputs": [],
   "source": [
    "def l2_penalty(w):\n",
    "    return tf.reduce_sum((w**2)) / 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.12.3.3 定义训练和测试\n",
    "\n",
    "下面定义如何在训练数据集和测试数据集上分别训练和测试模型。与前面几节中不同的是，这里在计算最终的损失函数时添加了$L_2$范数惩罚项。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 215,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-12T05:14:06.603709Z",
     "start_time": "2020-04-12T05:14:06.596727Z"
    }
   },
   "outputs": [],
   "source": [
    "def linreg(X, w, b):\n",
    "    return tf.matmul(X, w) + b\n",
    "\n",
    "def sgd(params, lr, batch_size, grads):\n",
    "    \"\"\"Mini-batch stochastic gradient descent.\"\"\"\n",
    "    for i, param in enumerate(params):\n",
    "        param.assign_sub(lr * grads[i] / batch_size)\n",
    "\n",
    "def set_figsize(figsize=(3.5, 2.5)):\n",
    "    \"\"\"Set matplotlib figure size.\"\"\"\n",
    "    plt.rcParams['figure.figsize'] = figsize\n",
    "\n",
    "def semilogy(x_vals, y_vals, x_label, y_label, x2_vals=None, y2_vals=None,\n",
    "             legend=None, figsize=(3.5, 2.5)):\n",
    "    \"\"\"Plot x and log(y).\"\"\"\n",
    "    set_figsize(figsize)\n",
    "    plt.xlabel(x_label)\n",
    "    plt.ylabel(y_label)\n",
    "    plt.semilogy(x_vals, y_vals)\n",
    "    if x2_vals and y2_vals:\n",
    "        plt.semilogy(x2_vals, y2_vals, linestyle=':')\n",
    "        plt.legend(legend)\n",
    "    plt.show()\n",
    "    \n",
    "def squared_loss(y_hat, y): \n",
    "    # 注意这里返回的是向量\n",
    "    return (y_hat - tf.reshape(y, y_hat.shape)) ** 2 / 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 220,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-12T05:17:52.894709Z",
     "start_time": "2020-04-12T05:17:52.885732Z"
    }
   },
   "outputs": [],
   "source": [
    "batch_size, num_epochs, lr = 1,100, 0.003\n",
    "net, loss = linreg, squared_loss\n",
    "train_iter = tf.data.Dataset.from_tensor_slices((train_features, train_labels)).batch(batch_size).shuffle(batch_size)\n",
    "\n",
    "def fit_and_plot(lambd):\n",
    "    w, b = init_params()\n",
    "    train_ls, test_ls = [], []\n",
    "    for _ in range(num_epochs):\n",
    "        for X, y in train_iter:\n",
    "            with tf.GradientTape(persistent=True) as tape:\n",
    "                # 添加了L2范数惩罚项,，广播机制使其变成长度为batch_size的向量\n",
    "                l = loss(net(X, w, b), y) + lambd * l2_penalty(w)\n",
    "            grads = tape.gradient(l, [w, b])\n",
    "            sgd([w, b], lr, batch_size, grads)\n",
    "        train_ls.append(tf.reduce_mean(loss(net(train_features, w, b),\n",
    "                             train_labels)).numpy())\n",
    "        test_ls.append(tf.reduce_mean(loss(net(test_features, w, b),\n",
    "                            test_labels)).numpy())\n",
    "    semilogy(range(1, num_epochs + 1), train_ls, 'epochs', 'loss',\n",
    "                 range(1, num_epochs + 1), test_ls, ['train', 'test'])\n",
    "    print('L2 norm of w:', tf.norm(w).numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.12.3.4 观察过拟合\n",
    "\n",
    "接下来，让我们训练并测试高维线性回归模型。当`lambd`设为0时，我们没有使用权重衰减。结果训练误差远小于测试集上的误差。这是典型的过拟合现象。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 219,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-12T05:14:38.580519Z",
     "start_time": "2020-04-12T05:14:34.048382Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQQAAAC1CAYAAAC56YFuAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAdlElEQVR4nO3deXxU9bn48c8zk8lGQggJYEiIAdxQkMWwWPWqbVVQW7VaLWhXb2l/v97W29+tVW5rl7v8rn313t+1/bW1rXVpr9adKqWUct2KG0uCQQICYc8ChIQkZF8mz/3jnCzEJEwgM2eSed6v17zIc+bMzDNnZh7O+Z7z/X5FVTHGGACf1wkYY6KHFQRjTDcrCMaYblYQjDHdrCAYY7pZQTDGdIvzOoFIyMzM1Ly8PK/TMCYqFBYWVqnqhP7ui4mCkJeXR0FBgddpGBMVROTgQPeNyEMGEblZRB4RkZdF5Fqv8zFmtIiagiAij4lIpYgU91m+WER2icgeEbkfQFVfUtUvA18A7vAgXWNGpagpCMATwOLeC0TED/wcWAJcCCwVkQt7rfJd935jzDCImoKgquuB430WLwD2qOo+VW0DngFuEsePgD+r6pb+nk9ElotIgYgUHDt27NQJvPrPsP7HPfFr/wLv/KwnfuUHsOHhnnjdd2HTIz3x2hVQ8HhPvOZe2PK7nnj1N6Ho6Z541dfh/ed74pf+NxSv7IlXLocdq5y/gx1OvHONE7c3O/HudU7cWu/Ee1514uZaJ973VydurHLiA287cf0RJz600Ynrypy4vNCJaw44cUWRE1fvdeIj7s7bsV1OfGyXEx/d7sTVe524osiJaw44cXmhE9eVOXHpJieuP+LEB9524sYqJ96/3omba5x4z6tO3FrvxLvXOXF7sxPvXOPEwXYn3rHKibsUr4Q//K+e+P3nne3fpehp5/PpsuW/4E/f6okLHnc+3y6bHnE+/y4bHna+H13e+Znz/eny1kPwxoM98fofn/xde+NBeOs/e+Iz/e6dgagpCAPIBkp7xWXusq8DHwduE5Gv9vdAVf21quarav6ECf02qJ7s+D6o6dXWcnQ7VO3qiY9sg6qSnvjw+1C9pyeuKILje3vF7/X8IMD5UdT2ev6yvvFmqOv1Vks3woly9810nhx3Bp24/rAbdzhxw1EnDrY7caNbCDtanbjJ/cF1tLhxtRO3N7ux+wNsa3Tillonbq134zonbjnhxifcuM6JW7viWidua3Tiphon7voBN1X3iaucuKPViRsqnbjrB95w1Ik7O5y4/rAbB924wom104lPlDtxl7oyZ/t2qT3obP/ecXmvuGa/8/l1Ob63pziCU/gOb+2Jq0qc70d3vMv5/nQ51ieu3OncuuMdPcUVzvy7dwYkmno7ikgesFpVZ7rxp4HrVPVv3fizwAJV/fqAT9KP/Px8tbMMxjhEpFBV8/u7L9r3EMqAKb3iHKDCo1yMGfWivSBsBs4VkakiEg98BljlcU7GjFpRUxBE5GngXeB8ESkTkbtVtQP4O+AvwAfAc6q6fbDnMcacvqi5UlFVlw6wfA2wJsLpGBOTomYPwRjjPSsIwPaKOp7edMjrNIzxnBUEYG3xEb77UjENrR1ep2KMp6wgAIumZRDsVDYf6HuhpDGxxQoCMC83nYBf2LCv2utUjPGUFQQgKd7P3CnpbNhrBcHENisIrkXTxrOtvI76lnavUzHGM1YQXIumZdCpWDuCiWlWEFzzzk4n3u9jwz4rCCZ2WUFwJQb8zM0dx1slVV6nYoxnrCD0cvUFE9lx+AQVtc1ep2KMJ6wg9PLxGZMAeHVnpceZGOMNKwi9TJ8whryMZF7ZcdTrVIzxhBWEXkSEj8+YxLt7q+0yZhOTrCD08bEZk2gLdvJWSQgDsxozylhB6GN+XjppSQH+st0OG0zssYLQR5zfx3UXTeK/dxylpT3odTrGRNSILAgiMk1EHhWRF8Lx/DdcPJmG1g7W77bDBhNbIl4QhjJl20DciVvuDleOH5meQXpygNXvHw7XSxgTlbzYQ3iCEKdsE5FZIrK6z21iuBMM+H0snnkWr3xghw0mtkS8IAxlyjZV3aaqN/a5hXTV0JCncuvjhlmTaWoL8ppdpGRiSLS0IQw0ZVu/RCRDRH4JzBWRFf2tM+Sp3PpYNG08mSkJvFxUPuTHGjNSRcsw7NLPsgHnmFPVaqDfOR2HS5zfxydnT+bJDQepa2onLTkQzpczJipEyx5CVE7ZdvPcybQFO/lzsTUumtgQLQUhKqdsm5WdxrTMMfzhPTtsMLHBi9OOI2bKNhHhpjnZbNx/nHLrEm1igBdnGZaqapaqBlQ1R1UfdZevUdXzVHW6qv5rpPMayC1znbbNl2wvwcSAaDlkiFq5GcnMz0tn5ZYyVAds5zRmVLCCEIJb5+Ww91gj75fVeZ2KMWFlBSEE11+cRXycj5VbyrxOxZiwsoIQgrGJAa69cBKrtlbQ1tHpdTrGhI0VhBDdekkONU3tvLbTxkkwo5cVhBBdcU4mE1MTeKHQDhvM6GUFIURxfh+fmpfD67uOUVnf4nU6xoSFFYQhuO2SHIKdysvveX5VtTFhYQVhCM6ZmMLc3HE8V1Bq1ySYUckKwhB9+pIplFQ2sNWuSTCjkBWEIbpxdhaJAR/PFZSeemVjRhgrCEM0NjHA9TOz+GNRBc1tNryaGV2iZYCUEeW2/BxWvlfO2u2HuWVujtfpmCFob2+nrKyMlpbRf6YoMTGRnJwcAoHQB/exgnAaFk3NIHd8Ms9tLrOCMMKUlZWRmppKXl4eIv0N1DU6qCrV1dWUlZUxderUkB9nhwynwecTPn1JDu/uq+ZgdaPX6ZghaGlpISMjY1QXA3DG8sjIyBjynpAVhNN0W34OPoHnC+zKxZFmtBeDLqfzPkdsQRCRMSJSKCI3evH6WWlJXHneBJ4vLKUjaB2ezOgwImduct0HPBeeLENzx/wpHD3RynqbKdqEqLa2ll/84hdDftz1119PbW1tGDI62YicuUlEPg7sADztevjRCyaRmRLPs5vtmgQTmoEKQjA4+CnsNWvWMG7cuHCl1S3iZxlUdb2I5PVZ3D1zE4CIdM3c9G/Ahw4JRORqYAxO8WgWkTWqGvH99vg4H7fOy+HRt/ZTWd/CxNTESKdgRpj777+fvXv3MmfOHAKBACkpKWRlZVFUVMSOHTu4+eabKS0tpaWlhXvuuYfly5cDkJeXR0FBAQ0NDSxZsoTLL7+cd955h+zsbF5++WWSkpKGJb9oOe3Y38xNCwdaWVW/AyAiXwCq+isGIrIcWA6Qm5s7nLme5Pb5U/jV+n2s3FLOV6+cHrbXMcPvh3/czo6KE8P6nBdOHsv3P3HRgPc/+OCDFBcXU1RUxBtvvMENN9xAcXFx96nBxx57jPHjx9Pc3Mz8+fO59dZbycjIOOk5SkpKePrpp3nkkUe4/fbbefHFF7nrrruGJf9oaVQc0sxN3SuoPqGqqwe474ymcgvV9AkpLMgbz7ObrcOTGboFCxacdJ3AT3/6U2bPns2iRYsoLS2lpKTkQ4+ZOnUqc+bMAeCSSy7hwIEDw5ZPSHsIInIP8DhQD/wGmAvcr6rrhimPqJy5KVR3zJ/CPzy/lY37j7NoWsapH2CiwmD/k0fKmDFjuv9+4403eOWVV3j33XdJTk7mqquu6vc6goSEhO6//X4/zc3DN2dIqHsIX1LVE8C1wATgi8CDw5ZFlM7cFKrrZ2WRmhhnjYvmlFJTU6mvr+/3vrq6OtLT00lOTmbnzp1s2LAhwtmFXhC6dumvBx5X1a30v5t/6icaQTM3hSop3s/Nc7JZs+0wdU3tXqdjolhGRgaXXXYZM2fO5N577z3pvsWLF9PR0cHFF1/MAw88wKJFiyKen4Ry3Csij+M0/E0FZgN+4A1VvSS86Q2P/Px8LSgoCOtrFJfXceP/f4sffOJCvnBZ6NeOm8j64IMPmDFjhtdpREx/71dEClU1v7/1Q91DuBu4H5ivqk1AAOewwbhmZqcxKzuNZ6xx0YxgoRaES4FdqlorIncB3wVsyKA+PrNgCjuP1FNUGv4ryowJh1ALwsNAk4jMBr4NHAR+F7asRqhPzp5McryfZzZZ46IZmUItCB3q7AffBPxEVX8CpIYvrZEpNTHAJ2dPZtXWCupbrHHRjDyhFoR6EVkBfBb4k9v3IPRhWGLIZxbk0tweZNXWEXMZhTHdQi0IdwCtONcjHME54/DjsGU1gs3OSWNG1lh+v/GQ16kYM2QhFQS3CDwFpLnjD7SoqrUh9ENEWLZgCtsrTrDNhmo3fZxu92eAhx56iKampmHO6GQhFQQRuR3YBHwauB3YKCK3hTOxkeymudkkBnz8fpPtJZiTRXtBCLW343dwrkGoBBCRCcArwAvhSmwkG5sY4BMXT2ZVUTnfuWEGKQnR0qnUeK139+drrrmGiRMn8txzz9Ha2sott9zCD3/4QxobG7n99tspKysjGAzywAMPcPToUSoqKrj66qvJzMzk9ddfD0t+obYh+LqKgat6CI+NSUsX5tLYFmRVkTUuRrXHb4D3nnL+DrY78dZnnbityYmLX3Tiljon3uF2s2msduJdf3bi+lOP1/Pggw8yffp0ioqKuOaaaygpKWHTpk0UFRVRWFjI+vXrWbt2LZMnT2br1q0UFxezePFivvGNbzB58mRef/31sBUDCP1HvVZE/iIiX3DHIPgTsCZsWY0Cc6eM44KzUnlmsx02mP6tW7eOdevWMXfuXObNm8fOnTspKSlh1qxZvPLKK9x33328+eabpKWlRSynkPZlVfVeEbkVuAynU9OvVfUPYc1shBMRli7I5furtlNcXsfM7Mh9qGYIvvinnr/9gZPj+OST48S0k+MxGSfHqZOG9NKqyooVK/jKV77yofsKCwtZs2YNK1as4Nprr+V73/vekJ77dIW826+qL6rq/1HVb1oxCM3N1rho+ujd/fm6667jscceo6GhAYDy8nIqKyupqKggOTmZu+66i29961ts2bLlQ48Nl0H3EESknv5HLhJAVXVsWLIaJdKSAtx48WRefq+cf7zeGhfNyd2flyxZwrJly7j00ksBSElJ4cknn2TPnj3ce++9+Hw+AoEADz/8MADLly9nyZIlZGVlha0dIaTuzyNdJLo/D2TLoRo+9Yt3+L+3zGLZwvCN7WhCY92fh6f7szlNXY2LT9thgxkBrCCEmYiwbGEu28rr7MpFE/VGZEEQkStE5Jci8hsRecfrfE7lpjnWuBhNYuEwGU7vfY7IqdxU9U1V/SqwGvhtOPMdDmlJzpWLLxeV09Da4XU6MS0xMZHq6upRXxS6poNPTBza5EFeNHs/AfyMXgOs9JrK7RqcIdk3i8gqnLEb/63P47/U66rJZcDfhjvh4bBsYS7PF5bxclE5dy482+t0YlZOTg5lZWUcOzb65+NMTEwkJydnSI8ZkVO5uevkAnXu8PD93R+RmZtCNWfKuO5u0csW5MbMlOTRJhAInDQxijlZtLQh9DeVW/YpHnM3zuQx/YrUzE2h6t0t+n1rXDRRKloKwpCnclPV76tq1Dco9nbT3GySAn4bPMVErWgpCCN6KrdQjU0McNMcZ8zFEzbmoolC0VIQRvRUbkNx58KzaW4P8tJ75V6nYsyHeHHacdRN5TYUs3LSuDgnjac2HBr1p77MyOPFWYalAyxfQ4yMsXDnwlzue3EbhQdryM8b73U6xnSLlkOGmPKJ2ZNJTYjjyQ0HvU7FmJNYQfBAcnwcn5qXzZriIxxvbPM6HWO6WUHwyLKFZ9PW0cmLhWVep2JMNysIHjn/rFTm56Xz1MaDdHZa46KJDlYQPHTnwrM5UN3E23urvE7FGMAKgqcWzzyL9OSADZ5iooYVBA8lBvzcdkkO67YfpbK+xet0jLGC4LXPLMilo1N5vsAaF433rCB4bPqEFBZNG88zmw9Z46LxnBWEKLBs4dmUHm9mfcnoH7TDRDcrCFHguosmkZkSz5MbrHHReMsKQhRIiPNzx/wpvLbzKGU14Z3u25jBWEGIEksXOMO82SlI4yUrCFEiJz2Zj14wiWc2ldLaEfQ6HROjrCBEkc9dejbVjW2s2XbY61RMjLKCEEUuPyeTaRPG8MQ71i3aeCPqC4KITBORR0XkhV7LxojIb0XkERG508v8hpPPJ3z+0jy2ltby3qEar9MxMSisBWGYZmnap6p391n8KeAFVf0y8MlhTttTt16SQ0pCHL9954DXqZgYFO49hCeAxb0X9JqlaQlwIbBURC4UkVkisrrPbeIAz5tDzzwOo6oFLiUhjlvnZbNmmw2eYiIvrAVBVdcDx/ss7p6lSVXbgK5Zmrap6o19bpUfelJHGU5RgBFw2DNUSxfm0hbsZOUW699gIsuLH9OQZmkSkQwR+SUwV0RWuItXAreKyMPAHwd43HIRKRCRgpE2j98FZ41lbu44ntlcaiMzm4jyYrLXIc3SpKrVwFf7LGsEvjjYi6jqr4FfA+Tn54+4X9XS+bl8+8X3bWRmE1Fe7CHExCxNZ+rG2VmkJsTx2Nv7vU7FxBAvCkLMzNJ0JpLj47j7iqms2XaEjfuqvU7HxIhwn3aM6VmaztRX/mY6k9MS+cEfdxC0sRJMBIT7LMNSVc1S1YCq5qjqo+7yNap6nqpOV9V/DWcOI1lSvJ9/vGEGHxw+wW/e3Od1OiYGjLpTdqPNDbOyWHzRWfxo7U7W7x5ZZ0vMyGMFIcqJCP9x+2zOm5TK3/1+C7uP1nudkhnFrCCMAGMS4njkc/kkBvwse2QDeyqtKJjwsIIwQkwZn8zvv7wIED7z6w28vccmdzHDzwrCCHLOxBSeWb6ItKQAdz26kX/64w6O1Nl8Dmb4SCxcGpufn68FBQVepzFsmto6+OfVO3h2cyl+n/A3507gyvMncNV5E8nNSPY6PRPlRKRQVfP7vc8Kwsh1qLqJ3717gHU7jnLouDM46/QJY1g0LYM5U8aRk55MRko8SQE/qqAoueOTERFUlU4Fv6+/K8nNaGYFYZQWhN72VzXy2s5K/rr7GFsO1tDQ2tHveudPSuXaiyaxtvgIB6ubmJM7jivPm8AnLp4c8t5Fa0eQOJ/PiskIZQUhBgpCbx3BTg4db+JwXQs1TW00twXxidDY1sHvNx5i55F6Zk8Zx9wp49h84DjbK04AkJmSQHZ6ElljExmbFEdlfSvHG9tobe8kqIqqUt3YRm1TOwBpSQFuvDiLG2ZlMX1iCmlJATpVOVzXwtETLeRljCErLZH2oNLU1kF7UElPDhDnt6YrL1lBiLGCMBhV5XhjGxkpCd3LymqaWFt8hJKjDZTVNnGkroUTLR1MGptAZkoCCXE+4nw+FGX8mHgmpSbSqbCvqoG1xUdo7egc8PXi/T7agj33pyUF+NiMiSyamsGsnDSmjE8m4Bf2VzXSEVTOmZhCW7CTA1WNCEJSvI+k+DjGJQUYk/DhzrlNbR1U1DZT3dBGTVMbVQ1tBPxC9rhkEgI+Gls72FPZwO6j9ZTVNNMRVC6cPJZ5Z6dzxTmZpI+JR1Vp7egk2Kkkx/vpVKhqaCU1MY7k+P47BLe0B/lz8WHSk+O58rwJiIycvSUrCFYQwqauuZ2tpbUcqG6kobUDnwgTUxOYkJrA/qpGymuaSU2MY0xCHH6fUFRay2s7K7v3MgBEoOtr6BMYqNtGZkoCF04ey6XTMiiraeL1nZVUhHiWJTMlvrv95IPDJ2hqCyICAd/JBcvvEwTo6FTi/T7y89LJTEmgraOTirpmapvayUpLZF9VI8fqWwGYPWUcs7LHAhDnc/Z+Glo7qGtup7qhlZz0ZK44N5N9VY1sOVhzUr+U5IQ4JqclMjYpQEKcj3MnpTIlPYmCAzXsPlqPAsFOpaXd2cuL8wsNLR20dnQyZXwSk8Ym4utVjPLz0vnI9MxBt4UVBCsIUaWzU9lf3UhxeR2H61poagsyfcIY/D5h95F6EuP9TJ+Qgl+EpvYgzW0dVDe2cbCqiS2HaiipbCA53s+V501gZnYaOelJZKYkkJ4cT0ZKPG0dnZTXNhPsVOLjfEzNHENmrz2iYKeytayWt0uqaGoPEvD7SAz48ItQ39JBpypZaYmU1jTz9p4qmtqCxPmErHFJpCUFOFLXTGpigC9dNpWymiYeeXMftU3t3T9eVSUlIY6xSQHSk+PZfbSe6sY24nzCzOw0xiT4u3Opb+mgoraFhtZ2Wjs66f1znJiaQJxP8PuFxDg/nap0dCpj4uMIxPkoPd70oWH2vnb1dO697oJBt78VBCsIo0pVQyspCXEkBvynXjkKBDuV3UfryU5PYmxiYMD12oOd7DpST+lxp7E3Ky0ppOfuTXBG7x7MYAXBixGTjDkjvf+3Hwn8PmFG1thTrhfw+5iZncbM7LQhPfdwsuZeY0w3KwjGmG5WEIwx3awgGGO6xcRZBhE5Bgw0g2omEE19iS2fwVk+gwsln7NVdUJ/d8REQRiMiBQMdArGC5bP4CyfwZ1pPnbIYIzpZgXBGNPNCoI73VsUsXwGZ/kM7ozyifk2BGNMD9tDMMZ0i+mCICKLRWSXiOwRkfs9eP0pIvK6iHwgIttF5B53+XgR+W8RKXH/TY9gTn4ReU9EVrvxVBHZ6ObyrDsfZ6RyGSciL4jITncbXerxtvmm+zkVi8jTIpIYye0jIo+JSKWIFPda1u/2EMdP3e/2+yIyL5TXiNmCICJ+4OfAEuBCYKmIXBjhNDqAf1DVGcAi4GtuDvcDr6rqucCrbhwp9+DMudnlR8B/urnUAHdHMJefAGtV9QJgtpuXJ9tGRLKBbwD5qjoT8ONMVBzJ7fMEsLjPsoG2xxLgXPe2HHg4pFdQd2isWLsBlwJ/6RWvAFZ4nNPLwDXALiDLXZYF7IrQ6+e4X6qPAqtxetNWAXH9bbMw5zIW2I/bztVruVfbJhsoBcbj9BJeDVwX6e0D5AHFp9oewK+Apf2tN9gtZvcQ6PmAu5S5yzwhInnAXGAjMElVDwO4/06MUBoPAd8GuoYQygBq1ZmxGyK7jaYBx4DH3UOY34jIGDzaNqpaDvw7cAg4DNQBhXi3fboMtD1O6/sdywWhv47knpxyEZEU4EXg71X1hEc53AhUqmph78X9rBqpbRQHzAMeVtW5QCORPXQ6iXtsfhMwFZgMjMHZLe8rWk7bndZnF8sFoQyY0ivOASoinYSIBHCKwVOqutJdfFREstz7s4DKCKRyGfBJETkAPINz2PAQME5EugbSieQ2KgPKVHWjG7+AUyC82DYAHwf2q+oxVW0HVgIfwbvt02Wg7XFa3+9YLgibgXPdVuJ4nAaiVZFMQJyheh8FPlDV/9frrlXA592/P4/TthBWqrpCVXNUNQ9nW7ymqncCrwO3RTIXN58jQKmInO8u+hiwAw+2jesQsEhEkt3PrSsfT7ZPLwNtj1XA59yzDYuAuq5Di0FFokEmWm/A9cBuYC/wHQ9e/3Kc3bj3gSL3dj3OsfurQIn77/gI53UVsNr9exqwCdgDPA8kRDCPOUCBu31eAtK93DbAD4GdQDHwX0BCJLcP8DRO+0U7zh7A3QNtD5xDhp+73+1tOGdHTvkadqWiMaZbLB8yGGP6sIJgjOlmBcEY080KgjGmmxUEY0w3KwjGcyJyVVfvSuMtKwjGmG5WEEzIROQuEdkkIkUi8it37IQGEfkPEdkiIq+KyAR33TkissHti/+HXv30zxGRV0Rkq/uY6e7Tp/Qa++Ap92pARORBEdnhPs+/e/TWY4YVBBMSEZkB3AFcpqpzgCBwJ04nny2qOg/4K/B99yG/A+5T1YtxrpTrWv4U8HNVnY3TF6Drctq5wN/jjE0xDbhMRMYDtwAXuc/zL+F9l8YKggnVx4BLgM0iUuTG03C6Sj/rrvMkcLmIpAHjVPWv7vLfAn8jIqlAtqr+AUBVW1S1yV1nk6qWqWonziXcecAJoAX4jYh8Cuha14SJFQQTKgF+q6pz3Nv5qvqDftYb7Fr4weYub+31dxBn0JEOYAFOb9CbgbVDzNkMkRUEE6pXgdtEZCJ0j+V3Ns53qKu33zLgLVWtA2pE5Ap3+WeBv6oz1kOZiNzsPkeCiCQP9ILuOBFpqroG53BiTjjemOkRd+pVjAFV3SEi3wXWiYgPp8fd13AGLrlIRApxRhG6w33I54Ffuj/4fcAX3eWfBX4lIv/kPsenB3nZVOBlEUnE2bv45jC/LdOH9XY0Z0REGlQ1xes8zPCwQwZjTDfbQzDGdLM9BGNMNysIxphuVhCMMd2sIBhjullBMMZ0s4JgjOn2P09Lrh3NhNEIAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "L2 norm of w: 13.024705\n"
     ]
    }
   ],
   "source": [
    "fit_and_plot(lambd=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.12.3.5 使用权重衰减\n",
    "\n",
    "下面我们使用权重衰减。可以看出，训练误差虽然有所提高，但测试集上的误差有所下降。过拟合现象得到一定程度的缓解。另外，权重参数的$L_2$范数比不使用权重衰减时的更小，此时的权重参数更接近0。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 202,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-12T05:04:42.226716Z",
     "start_time": "2020-04-12T05:04:36.905520Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAC1CAYAAACQyynAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAadklEQVR4nO2de3hV1Zn/P2+uh5BwC6BclIA4KKAgpAwU2uo4VkBRqx28FMfpT+XpzDhjx9GKT+t0cNqndmpba4sgVqpVf3hBWpXSQkHwUvESKCpXE0AkoBBAIAESSPLOH3sfzkk8Jznn5Nyy9/t5OE/Ou/Zaa79ns797rb2uoqoYhuE/cjLtgGEYmcHEbxg+xcRvGD7FxG8YPsXEbxg+xcRvGD4lL9MOpILevXtrWVlZpt0wjIyzdu3a/araJ9IxT4q/rKyMioqKTLthGBlHRHZGO+apar+ITBOR+YcPH860K4aR9XhK/Kr6sqrO7N69e6ZdMYysx1PiNwwjdjz1zi8i04BpQ4cObTticzM0NUB+l7T4ZWSOkydPUl1dTX19faZdSSmBQICBAweSn58fcxrx4sSe8vJyjdrgd+IY/KocxtwEF96dXseMtLNjxw5KSkooLS1FRDLtTkpQVQ4cOEBtbS2DBw9ucUxE1qpqeaR0/qv2FxTBqOtgwNhMe2Kkgfr6ek8LH0BEKC0tjbt246lqf8xc/F+Z9sBII14WfpBEfqOnSv64uvqOHYSaral3yvA1hw4d4uGHH4473dSpUzl06FAKPArhKfHH1dX35FXw8u2pd8rwNdHE39TU1Ga6pUuX0qNHj1S5Bfi12g9wyX1QWJJpLwyPM2vWLLZt28bo0aPJz8+nuLiYfv36sX79ejZt2sRVV13Frl27qK+v5/bbb2fmzJlAaJRqXV0dU6ZMYdKkSbz55psMGDCAF198kS5dOt5T5V/xD7kw0x4YaWb2yxvZtOdIUvMc3r8b3582Iurx+++/nw0bNrB+/XpWr17NZZddxoYNG061yi9YsIBevXpx/PhxvvCFL3DNNddQWlraIo/KykoWLlzIo48+yvTp03nhhReYMWNGh333VLU/bvZuhPeezbQXho8YN25ci+64hx56iFGjRjF+/Hh27dpFZWXl59IMHjyY0aNHAzB27Fg++uijpPjiqZI/5kE+QdY9Cet+CyOugrzClPpmZJ62Suh00bVr11PfV69ezYoVK1izZg1FRUVceOGFEbvrCgtD92Zubi7Hjx9Pii+eKvnjHts/6dvw7Q9M+EbKKCkpoba2NuKxw4cP07NnT4qKitiyZQtvvfVWWn3zVMkfNyWnZ9oDw+OUlpYyceJERo4cSZcuXTjttNNOHZs8eTLz5s3j/PPPZ9iwYYwfPz6tvvlveG9rPn4b1j0B0x6CXH8/C73I5s2bOffcczPtRlqI9FtteG9b1H4ClX+GQ1HXPDAMT2JF3TmXw7lXQI49Bw1/YeK3qr7hUzxV3CW8jNfudTBvEuzbnBrHDCML8ZT4E17Gq+R0yO8KDXWpccwwshCr8wJ06w83L8u0F4aRVjxV8neYppNw0tvLPRnpJdEpvQAPPvggx44dS7JHIUz8QQ7vhv89C963sf5G8shm8Vu1P0i3/jDmRug7PNOeGB4ifErvJZdcQt++fXnuuedoaGjga1/7GrNnz+bo0aNMnz6d6upqmpqauPfee9m7dy979uzhoosuonfv3qxatSr5zqmq5z5jx45Vw1BV3bRpU8uABVNV1z3lfG884djrn3HshqOO/cEixz5+yLE3vujYdfsde8tSxz7yabvn37Fjh44YMUJVVZctW6a33nqrNjc3a1NTk1522WX66quv6qJFi/SWW245lebQoUOqqjpo0CCtqalJ/LeqKlChUXSS9dV+ERkiIo+JyKK0nPDwbji4PS2nMvzF8uXLWb58ORdccAFjxoxhy5YtVFZWct5557FixQruvvtuXn/9ddK26Uy0p0IqP8ACYB+woVX4ZGArUAXManVsUaz5J1zyNzWq3j9IdfG3EktvZB2RSsN0El7y33HHHTpv3ryI8Q4cOKBPPvmkTpw4UWfPnq2q3i35H3eFfgoRyQXmAFOA4cD1IpLeF/CcXLjyYfjSHWk9reFdwqf0XnrppSxYsIC6Omc8ye7du9m3bx979uyhqKiIGTNmcOedd7Ju3brPpU0FGWnwU9XXRKSsVfA4oEpVtwOIyDPAlcCmtDp3ztS0ns7wNuFTeqdMmcINN9zAhAkTACguLuapp56iqqqKu+66i5ycHPLz85k7dy4AM2fOZMqUKfTr1y8lDX4Zm9Lrin+Jqo507a8Dk1X1Fte+Efhb4PvAD4FLgF+r6o+i5DcTmAlw5plnjt25swOz9Ha+6Sztfe7liedhZAU2pTf6lN5s6uqLtOuAquoB4FvtJVbV+cB8cObzd8iTN34On+008RueJpvEXw2cEWYPBPbEk0Hca/hF47KfQpdeHcvDMLKcbOrqexc4W0QGi0gBcB3wUjwZaAwTe040NvPxgWMcbWiMnlGPM6GwOJ5TG0anIyPiF5GFwBpgmIhUi8jNqtoI3AYsAzYDz6nqxjjzbXdKb+W+Wr78k1W8UbW/7cy2/hH+cGc8pzeylEy1a6WTRH5jplr7r48SvhRY2oF8XwZeLi8vvzVanJJCZ//yuvo2Sn6A/ZVQtQIaam1nn05MIBDgwIEDnt6pV90tugOBQFzpsumdv8PE8s5fHHB+cl1b1X6A8f8MX/w38OgN4xcGDhxIdXU1NTU1mXYlpQQCAQYOHBhXGk+JP5aSv2thLhCD+HPzk+makSHy8/Nb7JBjhMimBr+0UJiXS0FuDrXtVfsBqlbCQ2Ogbl/qHTOMNOMp8ce6hl9xIK/t1v5TEU+DXoPheGr3STeMTOAp8cfS1QdQXJjXfrUf4PSRMOMF6PM3SfLQMLIHT4k/VooL82Kr9gdpqIOTydkc0TCyBU+JP55qf13DydgyPbgdfnIWbFicBA8NI3vwlPhjrfaXxFrtB+g5GCbcBv1GJcFDw8gePNXVFyvFgTy21cQofhG4+N7UOmQYGcBTJX/M1f54Sv4g+yvhk/c74J1hZBeeEn/Mrf2BOBv8VOGpq2HlfR300DCyB39W+wvyaGhs5kRjMwV5MTz/RODqR53ZfobhEfwpfnd8/9GGRgryCmJLdOb4FHpkGOnHU9X+WCkujHFyT2s++gu89kAKPDKM9OMp8cfa4FcS68y+1ux4Dd562HbzNTyBp8Qf+/Bed05/vOL/4m1wxxZb5cfwBL5+5293QY/W2KIehofwVMkfK8F3/tp4S36ATzfAoxfD3rhWGDOMrMOX4i9JtOQHZ5pvUwPUt92uYBjZjj+r/ada+2Oc3NMicR/41htJ9sgw0o8vS/6iglxEEiz5gzQ12go/RqfGU+KPtatPRCguyEvsnT/IY38PL/1b4ukNI8N4SvyxdvWBO6e/IyX/3/4zjLkp8fSGkWFiEr+I3C4i3cThMRFZJyJfTbVzqSShmX3hjLrWdvQ1OjWxlvz/T1WPAF8F+gDfBO5PmVdpwFnNpwPiB6fF/+350NiQHKcMI43EKv7gzhVTgd+o6ntE3lW309Dhkh9g9zr4412w7ZXkOGUYaSTWrr61IrIcGAzcIyIlQHPq3Eo9JYE8Pj1c37FMhlzodPudfl4yXDKMtBKr+G8GRgPbVfWYiPTCqfp3WpJS8ouEhK9qW3sZnYpYq/0TgK2qekhEZgDfAzr1ELfiwvyOtfaHs2YOLLwuOXkZRpqIVfxzgWMiMgr4DrAT+G3KvApDRLqKyBMi8qiIfCNZ+RYH8qg70UhzcxK2b84tgLyAre1vdCpiFX+jOhuAXwn8QlV/ASQ8xU1EFojIPhHZ0Cp8sohsFZEqEZnlBl8NLFLVW4ErEj1na0oK81CFYyebOp7ZuFth+hOQ36XjeRlGmohV/LUicg9wI/AHEckFOrKN7ePA5PAAN885wBRgOHC9iAwHBgK73GhJUKpD1+DMvvoExvdH4/BuqK5IXn6GkUJiFf+1QANOf/+nwADgJ4meVFVfAw62Ch4HVKnqdlU9ATyDU9OoxnkAxONvu3Tr4oj/yPEkvfcDPH8T/P5fnMY/w8hyYhKTK/inge4icjlQr6rJfucfQKiEB0f0A4DFwDUiMhd4OVpiEZkpIhUiUlFTU9Puybp3cSouR5JZ8l/2U/jG89bqb3QKYurqE5HpOCX9apzBPb8UkbtUdVESfYmkGFXVo8TQraiq80XkE2BaQUHB2PbiB8V/+FgSxR++pVf9EQh0S17ehpFkYq1Gfxf4gqrepKr/iFNFT/YeVtXAGWH2QGBPPBnEM7GnW8AV//Ekij/IKz+E+V+xhT6NrCbWQT45qho+ef0AyZ8R+C5wtogMBnYD1wE3xJOBiEwDpg0dOrTduCmp9gcZ8hVoOuF0ARpGlhKrgP8kIstE5J9E5J+APwBLEz2piCwE1gDDRKRaRG5W1UbgNmAZsBl4TlXjWigvnpI/uJRXSkr+sklwyWzIK4CGWmsANLKSmEp+Vb1LRK4BJuK8m89X1d8lelJVvT5K+FI68FCJh7zcHIoL81Ij/iDHDsJjX4WzLoKpCXeOGEZKiHkNP1V9AXghhb50mHiq/eBU/ZPa1deaLj1h1HVw5oTUncMwEqTNar+I1IrIkQifWhE5ki4nYyWeaj9Aty75qS35ReDLd0LZRMd+ay6s/B97DTCygjZLflXtVLtUxFvydwvkcSSV4m/Nvk1w9EBoHMDHb0OfYdClR/p8MAwX367hB061P6Ulf2uu+CVMd8dG1R+GJy6Hvzzo2E2N8MoPYM9fHbvxBOzd5IwXAGhudlYMslqDkSQ8Jf546d4lPzVdfW2R61a28rvCjMUw2p2oeKQaXv+psyMQwJHdMHcCbFni2Acq4Qd9YeNix/50A/zgdNj6R8fesx5+PBi2rXLs6gp4YBjsXOPYH78FPx8J1Wsde8fr8NAFofNtXw2/Ggc1Hzp21QqYOxEObnfsD5fBI19x5i+Ac95fXxJavnzzEvjNVDj+mWNvegmeuMLp7QDY+Dt46prQkmcfLIL/fy00u9M13n8enr0xdJ3eexYWzwyzn2m5WvJfn4Y/3BlmPwXLvhuy1/0WVt4Xstc+DqvDVp6rWACv/yxkv/sYvPmrkP3Oo/D2Iy3tigUh++35sO7JkP3WPMfHU/Zc5zcGWTMHNr0Yst/8FWwJa9t+85fw4fKQ/ZdfQNXKkP3Gg7D91ZD9+s9CBUOCeEr8sS7dHSTl7/xtkZsHg78Evc927J5l8L0aOH+6Y3ftDf/wOAxy2wuKSuHv7oW+I1y7F4y7BXoMcuwuPWHk1VByumMHesDffNWJBxDoDmVfcv6CM/qw/xgoKHLsghLoew7kFTp2flcn7+BYhbxCKO4LObmOnZPnpJXwWyhskGbzSWisD4WdrHd6P4I1lxN1cCRsDNfxg3BwR8iu2+u8JgU5tMt5wAX5bAfsDptEtW8zfLwmZH/6gfOAC7J7bejBCM5DMVxcO16FymUhu2oFfPinkL1lCWwNszcuDj14Ad5bCFvDxLz2iZb2O4+2FPuaOS2Pv/Fgy/O99oDjQ5DVP4JtYf6+8j/Q0DHxi3qwGlleXq4VFe3PrntoZSU/+/OHVP5wCvm5nnoOGp2B8NWfmt1V8XLc+7Cp0TkWfNg2nnAetMGaY2OD82BuZx6JiKxV1fJIx3x9x58a5Zep0t/wN+HCzckJCR8ckQeFD86Asdyw9vm8wg5PIDPxk6JRfoaR5XhK/PG/86dwiK9hZDmeEn8iXX0AR5K1kKdhdCI8Jf54sWq/4Wd8Lf6Uzuk3jCzH3+K31n7Dx3hK/PE2+AXycynMyzHxG77EU+KPt8EPMjzKzzAyiKfEnwgZGd9vGFmAid9KfsOn+F783QIpXsrLMLIU34vfSn7Dr/he/D2KCjh01MRv+A9PiT/erj6APiWF1DY0Up+M3XoNoxPhKfEn0tXXp9hZvKKmtiFVbhlGVuIp8SdC7xJnpZr9dSZ+w1+Y+K3kN3yK78Xfp8QR//66Exn2xDDSi+/FX9o1KH4r+Q1/4XvxF+Tl0L1LvlX7Dd+R9eIXkSEi8piILGo/dmL0KSm0kt/wHSkVv4gsEJF9IrKhVfhkEdkqIlUiMqutPFR1u6renEo/excXmPgN35Hqkv9xYHJ4gIjkAnOAKcBw4HoRGS4i54nIklafvin2D4A+JQGr9hu+I+YtuhNBVV8TkbJWweOAKlXdDiAizwBXquqPgMtT6U80nJLfWvsNf5GJd/4BwK4wu9oNi4iIlIrIPOACEbmnjXgzRaRCRCpqamricqh3cSF1DY0cP2FDfA3/kNKSPwqRthmJumeYqh4AvtVepqo6H5gPznZd8TgU6utv4IxeRfEkNYxOSyZK/mrgjDB7ILAnSty4SGRiD4SN77dGP8NHZEL87wJni8hgESkArgNeSkbGiUzsgbCS3xr9DB+R6q6+hcAaYJiIVIvIzaraCNwGLAM2A8+p6sYknS+hkr+3lfyGD0l1a//1UcKXAksjHevg+V4GXi4vL781nnSlxe7Mvlpr8Tf8Q9aP8IuHREv+/NwcehblU1NXnyLPDCP78JT4E33nB+e9f98Rq/Yb/sFT4u8IZ/UppnJfXabdMIy04SnxJ1rtBxg5oDs79h+l1jbwMHyCp8TfkWr/iP7dANi450iy3TKMrMRT4u8II/o7D4wNu+OvNRhGZ8RT4u9Itb9PSSGndwtYyW/4Bk+JvyPVfoCRA7pZyW/4Bk+Jv6OM6N+dbTV1HDvRmGlXDCPleEr8Han2g9Pi36yw+ZPaJHtmGNmHp8SfjGo/wNqdB5PplmFkJZ4Sf0c5vVuA8kE9mbNqG3uP2FBfw9uY+MMQEf736+fT0NjErBfeRzWuNUEMo1ORiZV8spohfYr5zqXncN+STUz68SouPrcvp3UL0C2QR15uDrk5Qo4IOQIiIAjirk2UI9IiTMA9Fm7LqfBgXPcftD7uhgbTBiOFn7NFuhb5hI62jkt4mhbxW56zNa3DIsWNKc7ns45wvs/HiuRT9PyiRI4SP9ZztcwnlpwSz789cnOE/j26JJzeU+IXkWnAtKFDh3Yon29OLKO0uIDf/3U3z1dUc9y27zaykDN7FfHady5KOL14sWpbXl6uFRUVScuv/mQTtfWNNDY309SsNDeDojQrqCoKaOvvqBsW+k7rY4TSBI8RdoxW+Yand4zQn/Dwluk1PGqIVvFb/A2PFiV96LbRCGEtj4SHa4TlGqOlaxkn9vu0raiRzh9P+njiRPchOXQtyGXKef3ajCMia1W1PNIxT5X8qSKQn0sgPzfTbhhGUrEGP8PwKSZ+w/ApJn7D8CkmfsPwKZ5s7ReRGmBnlMO9gf1pdKc9zJ/oZJMv0Dn9GaSqfSId8KT420JEKqJ1fWQC8yc62eQLeM8fq/Ybhk8x8RuGT/Gj+Odn2oFWmD/RySZfwGP++O6d3zAMBz+W/IZh4CPxi8hkEdkqIlUiMisD5z9DRFaJyGYR2Sgit7vhvUTkzyJS6f7tmWa/ckXkryKyxLUHi8jbrj/Putuop8uXHiKySES2uNdpQiavj4j8h/t/tUFEFopIIJ3XR0QWiMg+EdkQFhbxeojDQ+79/b6IjGkvf1+IX0RygTnAFGA4cL2IDE+zG43Af6rqucB44F9dH2YBK1X1bGCla6eT23G2Sg/yY+Dnrj+fATen0ZdfAH9S1XOAUa5fGbk+IjIA+HegXFVHArnAdaT3+jwOTG4VFu16TAHOdj8zgbnt5q6qnv8AE4BlYfY9wD0Z9ulF4BJgK9DPDesHbE2jDwPdG+jvgCU461zsB/IiXbcU+9IN2IHbDhUWnpHrAwwAdgG9cGa/LgEuTff1AcqADe1dD+AR4PpI8aJ9fFHyE/qPDFLthmUEESkDLgDeBk5T1U8A3L990+jKg8B3gGbXLgUOqWpw7fJ0XqchQA3wG/c15Nci0pUMXR9V3Q08AHwMfAIcBtaSuesTJNr1iPse94v4Iy2alJFuDhEpBl4Avq2qGdseSEQuB/ap6trw4AhR03Wd8oAxwFxVvQA4SvpfgU7hvktfCQwG+gNdcarWrcmW7rK4/+/8Iv5q4IwweyCwJ91OiEg+jvCfVtXFbvBeEennHu8H7EuTOxOBK0TkI+AZnKr/g0APEQku8pLO61QNVKvq2669COdhkKnr8/fADlWtUdWTwGLgi2Tu+gSJdj3ivsf9Iv53gbPdltoCnIabl9LpgDgrSj4GbFbVn4Udegm4yf1+E05bQMpR1XtUdaCqluFcj1dU9RvAKuDrGfDnU2CXiAxzgy4GNpGh64NT3R8vIkXu/13Qn4xcnzCiXY+XgH90W/3HA4eDrwdRSUfjSTZ8gKnAh8A24LsZOP8knGrY+8B69zMV5z17JVDp/u2VAd8uBJa434cA7wBVwPNAYRr9GA1UuNfo90DPTF4fYDawBdgAPAkUpvP6AAtx2htO4pTsN0e7HjjV/jnu/f0BTi9Fm/nbCD/D8Cl+qfYbhtEKE79h+BQTv2H4FBO/YfgUE79h+BQTv5FWROTC4AxCI7OY+A3Dp5j4jYiIyAwReUdE1ovII+68/zoR+amIrBORlSLSx407WkTecueR/y5sjvlQEVkhIu+5ac5ysy8Om7f/tDuCDhG5X0Q2ufk8kKGf7htM/MbnEJFzgWuBiao6GmgCvoEzuWWdqo4BXgW+7yb5LXC3qp6PM7osGP40MEdVR+GMiw8ON70A+DbO2gpDgIki0gv4GjDCzecHqf2VhonfiMTFwFjgXRFZ79pDcKb+PuvGeQqYJCLdgR6q+qob/gTwZREpAQao6u8AVLVeVY+5cd5R1WpVbcYZ5lwGHAHqgV+LyNVAMK6RIkz8RiQEeEJVR7ufYar63xHitTU2PNIU0yANYd+bcBbHaATG4cx6vAr4U5w+G3Fi4jcisRL4uoj0hVPrxg3CuV+CM9puAN5Q1cPAZyLyJTf8RuBVddYqqBaRq9w8CkWkKNoJ3XUOuqvqUpxXgtGp+GFGiLz2oxh+Q1U3icj3gOUikoMzq+xfcRbYGCEia3FWtrnWTXITMM8V93bgm274jcAjInKfm8c/tHHaEuBFEQng1Br+I8k/y2iFzeozYkZE6lS1ONN+GMnBqv2G4VOs5DcMn2Ilv2H4FBO/YfgUE79h+BQTv2H4FBO/YfgUE79h+JT/AwTgC46OhnzVAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "L2 norm of w: 0.3192926\n"
     ]
    }
   ],
   "source": [
    "fit_and_plot(lambd=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.12.4 简洁实现\n",
    "\n",
    "请查阅：https://www.tensorflow.org/tutorials/keras/overfit_and_underfit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-12T05:05:36.814471Z",
     "start_time": "2020-04-12T05:05:36.808487Z"
    }
   },
   "outputs": [],
   "source": [
    "def fit_and_plot_tf2(wd, lr=1e-3):\n",
    "    model = tf.keras.models.Sequential()\n",
    "    model.add(tf.keras.layers.Dense(1, \n",
    "                                    kernel_regularizer=regularizers.l2(wd), \n",
    "                                    bias_regularizer=None))\n",
    "    model.compile(optimizer=tf.keras.optimizers.SGD(lr=lr), \n",
    "                 loss=tf.keras.losses.MeanSquaredError())\n",
    "    history = model.fit(train_features, train_labels, epochs=100, batch_size=1, \n",
    "              validation_data=(test_features, test_labels),\n",
    "             validation_freq=1,verbose=0)\n",
    "    # model.summary()\n",
    "    train_ls = history.history['loss']\n",
    "    test_ls = history.history['val_loss']\n",
    "    # print(test_ls)\n",
    "    semilogy(range(1, num_epochs + 1), train_ls, 'epochs', 'loss',\n",
    "                     range(1, num_epochs + 1), test_ls, ['train', 'test'])\n",
    "    print('L2 norm of w:', tf.norm(model.get_weights()[0]).numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 204,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-12T05:05:47.170420Z",
     "start_time": "2020-04-12T05:05:39.610893Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQQAAAC1CAYAAAC56YFuAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3dd3zUdbb/8ddJISEhQCogARKQLgiEugqCBftaVvGKuLtexW3qlmvjurte9+fede+67uquiijYxUVWLIiCKFVpCZ1QAoSSAGkQ0tvk/P74TgqYhCQkMynn+XjMg7y/mZnvme9MDvOtH1FVjDEGwMfbBRhjWg5rCMaYStYQjDGVrCEYYypZQzDGVLKGYIyp5OftAjwhIiJCY2JivF2GMS1CQkJCpqpG1vS7dtEQYmJiiI+P93YZxrQIInK4tt/ZKoMxppI1BGNMJWsIxphK1hCAj7ak8sB7m71dhjFeZw0ByMgt5sCO9WQd3lk1MesAZB85M59OOSunVuXM/ZBzrFpOgpzj1WayD3JPVMt7ITetKqfvhrx052dVd85wcnm5k/Mz3dnlzllOdpU5ueCkO5c6ufCUk8tK3DnbnYudXHTayaVF7pzjzoVOLs51ckmBO+e5c76TS/KdXJznzgVOLspxcmmhO5925yInF2Y7uazYnU+5c4mTC0462VV6Vi5zcn6Wk8td7pzp5IoT9fIynFwhLx3S91Tl3DRn+VfmE877UyHnmPP+VTid6ry/lTnFef8rZB89Kx+Bkwer8qnDcDK5Wj7k3CqcTHbuU5kPOs9ZoaGfvfNgDQEYHRPKH/3nUv7Zo1UTP/gRLKmW358OX8yqyu/eBsufrMpv3wxfP12V37geVj1TledeBWueq8pzpsC3L1Tl2ZfChtlV+aXxED/X+bm81Mmb33RySb6Tt813ctFpJ+9Y6OT8DCcnfuzk3GNO3rvEydlHnJz0pZOz9jv54Eonp+928qFvnHxiu5NTNjo5NcHJx7Y4+eh6J6e5G+qhtU6u+KM7sMLJFX8kScucXPEh3/OZk/PcDTPxIycXuBve9gVOLnY3rK3vOrmi4SS87uRyd8PY9KqTK6x/CV6ZWJW/eR5evaIqr34W5l1dlVf+Cd68sSp//f/gnVuq8pe/h/emVeUvHod/3V2VP3sYPrinKn/6ECz6SVX+6Ofw8QNV+cOZsPhXVXnBj2DJI1X5/bsa9tk7H6ra5m9xcXFal+JSl/7giX/oq/M/qJp4YIXq4XVVOelL1cPrq/K+ZapHN1XlvV+opsRX5T1LVFM3V+XET1WPba3Kuz5WPb69Ku9cpHpip/Nzebnqzg9V0xKd7CpzcvoeJ5eVODkjycmlRU7O3O/kkgInZx10v8A8J5885OSiHCefOuLkglNOzk5xcn6Wk08fc3JeppNz05ycm+7kvAwn55xwcn6Wk08fc3LBSSdnpzi5MNvJpw47uSjHySeTnVyc5+SsA04uKXBy5n4nlxY7OWOfk8tKnZy+x8kul5PTEp1c4cRO1V0fVeVj21QTP6nKqVtUdy+uyikJzvtX4Wi88/5WOLLRef8rHF6vmrS8Kh/6VnX/11U5eY3qgZVV+eAq1YOrq/KBFarJa6vy/q+c56jQ0M/eOQDxWsvfimg7uB7C6NGj9VzHIdzxyjoKS1188sClHqrKGO8QkQRVHV3T72yVwW1MTBi7juVQUFLm7VKM8RprCG5xMaG4ypWtR7K9XYoxXmMNwW1U71BEYNOhU94uxRivsYbg1qWjPwO7hRB/+KS3SzHGa6whVDMuNoz4Q6coKSv3dinGeEWrbAgi0ldE5orIwqZ83gn9wiksdbE9xbYjmPbJ4w1BROaJSLqI7Dxr+jUisldE9ovI43U9h6oeVNV7m7q2sbHhAKw7kNXUT21Mq+CNbwhvANdUnyAivsCLwLXAEOBOERkiIsNEZPFZt6jmKiwsuAODuoewPtkagmmfPH6BFFVdLSIxZ00eC+xX1YMAIvI+cJOq/gm4wZP1TegXznsbjlBc5iLAz9eTszbG61rKNoSeQLWzOUhxT6uRiISLyGxgpIjMquU+94tIvIjEZ2Rk1LuQCX3DKS4rt+MRTLvUUhqC1DCt1mOqVTVLVX+qqv3c3yJqus8cVR2tqqMjI2u8fFyNxsWGIwLrDtpqg2l/WkpDSAF6VcvRQNOcz9lAXYL8GdazC2uTMr0xe2O8qqU0hE1AfxGJFZEOwH8An3irmIn9I9hyNJucolJvlWCMV3hjt+N8YB0wUERSROReVS0DHgCWAruBBaq6y9O1VZjUPxJXufLtfvuWYNoXb+xluLOW6UuAJR4up0aj+oTSKcCPVfsyueaiHt4uxxiPaSmrDC2Kv68PE/qFs3pfBu3hehHGVLCGUItJ/SNIzS4kOTPf26UY4zHWEGoxaYCzq3KtbUcw7Yg1hFr0DguiR5dANhy006FN+2ENoRYiwrjYMDYkZ9l2BNNuWEOow/i+4WTmlXAgw7YjmPbBGkIdxvV1Todeb4cxm3bCGkIdYsKD6NY5gA3Jth3BtA/WEOrgbEcIZ/1B245g2gdrCOcwvm84GbnFdjyCaResIZzD2NgwADYdstUG0/ZZQziHfpHBhAd3YGOyjddg2j5rCOcgIoyOCWXjIdvTYNo+awj1MCYmjKMnCzlxusjbpRjTrKwh1EPFdoSNth3BtHGttiGISLCIJIhIs1+VeUiPzgR38GWTHY9g2rhWOVCL22PAguap8kx+vj6M6hPKRmsIpo1rlQO1iMiVQCKQ5qmix8WGsTctl1P5JZ6apTEe1yoHahGRKUAwTvMoFJElqtqsI7RWnNew8dBJrh7avTlnZYzXeLwh1KKmgVrG1XZnVX0CQER+DGTW1AxE5H7gfoDevXufd4HDo7sQ4OfDhoPWEEzb1VI2KjZooJbKO6i+oaqLa/ldowZqqU2Any9xfULtzEfTprWUhtBiBmqpy7jYcHafyOF0gY3XYNqmltIQWtRALbUZ1zcMVTuvwbRdNlBLA4zo1ZUOfj5ssOHiTRtlA7U0QKC/L6N6d+Wb/dYQTNvUUlYZWo2J/SNJPJ5DRm6xt0sxpsm1lN2OrcZlAyL5y9K9rEnK4NZR0d4uxzRQaWkpKSkpFBW1/RPVAgMDiY6Oxt/fv96PsYbQQEN6dCY8uAOr91lDaI1SUlIICQkhJiYGkZr2drcNqkpWVhYpKSnExsbW+3G2ytBAPj7CxP4RrEnKpLzcrrPY2hQVFREeHt6mmwE41/EIDw9v8DchawiNMGlAJFn5JSQez/F2KaYR2nozqNCY12kNoREm9neOfFy1L8PLlRjTtKwhNEJkSABDenRmtTUE00DZ2dm89NJLDX7cddddR3Z2djNUdCZrCI00aUAkCYdPkVdc5u1STCtSW0NwuVx1Pm7JkiV07dq1ucqqZA2hkSYNiKCsXFl3wA5SMvX3+OOPc+DAAUaMGMGYMWOYMmUK06dPZ9iwYQDcfPPNxMXFMXToUObMmVP5uJiYGDIzMzl06BCDBw9m5syZDB06lKlTp1JYWNhk9dlux0aK6xNKUAdfVu/L4Koh3bxdjmmEpz7dReKxpt0wPOSCzjx549Baf//MM8+wc+dOtm7dysqVK7n++uvZuXNn5a7BefPmERYWRmFhIWPGjOEHP/gB4eHhZzxHUlIS8+fP59VXX2XatGn8+9//ZsaMGU1SvzWERgrw82V833BWJ9l2BNN4Y8eOPeM4gRdeeIFFixYBcPToUZKSkr7TEGJjYxkxYgQAcXFxHDp0qMnqqVdDEJFfAq8DucBrwEjgcVVd1mSVtEKT+kfw9Z50Dmfl0yc82NvlmAaq639yTwkOrvrcrFy5kuXLl7Nu3TqCgoKYPHlyjccRBAQEVP7s6+vbpKsM9d2G8J+qmgNMBSKBe4BnmqyKVuqygVEAtrfB1FtISAi5ubk1/u706dOEhoYSFBTEnj17WL9+vYerq/8qQ8URDtcBr6vqNmkvR3fUISY8iF5hHVm1L5O7J8R4uxzTCoSHh3PJJZdw0UUX0bFjR7p1q9r+dM011zB79myGDx/OwIEDGT9+vMfrq29DSBCRZUAsMEtEQoBmvahpayAiTOofyUdbUikpK6eDn+20Mef23nvv1Tg9ICCAzz//vMbfVWwniIiIYOfOqhEMHn744Satrb6f4HuBx4ExqloA+OOsNniFiEwUkdki8pqIfOutOsA5+zG/xEXCYRsM1rR+9W0IE4C9qpotIjOA3wKnGzPDphioRVXXqOpPgcXAm42po6lM6BeOn4/Y3gbTJtS3IbwMFIjIxcCjwGHgrUbO8w3Oc6CWag+dDsxvZB1NIiTQn7g+oazaaw3BtH71bQhlqqrATcDzqvo8ENKYGarqauDsq5RWDtSiqiVAxUAtO1T1hrNu6QAi0hs47d778R0icr+IxItIfEZG8/6xThrgXEUpPbftX3TDtG31bQi5IjILuBv4zP0/ev0vw3JuNQ3U0vMcj7kX59iIGjX1uAx1mTzQffajfUswrVx9G8IdQDHO8QgncP5Y/9KEdTR4oBZVfVJVvbpBscKQHp2JCglgpR2PYFq5ejUEdxN4F+jiHn69SFUbuw2hJq1ioJbaiAiTB0ayel8GZa52vzfW1KGxpz8D/P3vf6egoKCJKzpTvRqCiEwDNgK3A9OADSJyWxPW0SoGaqnLlIFR5BaVsflI85+zblqvlt4Q6ntg0hM4xyBUbNCLBJYDCxs6Q/dALZOBCBFJAZ5U1bkiUjFQiy8wryUO1FKXS/pH4OcjrNibztjYMG+XY1qo6qc/X3XVVURFRbFgwQKKi4u55ZZbeOqpp8jPz2fatGmkpKTgcrn43e9+R1paGseOHWPKlClERESwYsWK5ilQVc95A3aclX3OntaSb3FxceoJ02Z/q1f/bZVH5mUaJzEx8cwJ865T3fyO83NZiZO3vu/k4nwn71jo5MJsJ+/62Ml5mU7es8TJOSfOOf/k5GQdOnSoqqouXbpUZ86cqeXl5epyufT666/XVatW6cKFC/W+++6rfEx2draqqvbp00czMjLO7/WqKhCvtfyt1Hej4hcislREfuwegv0zWvgoS94weWAUe07kkpZjux/NuS1btoxly5YxcuRIRo0axZ49e0hKSmLYsGEsX76cxx57jDVr1tClSxeP1VSvVQZVfUREfgBcgrNHYI6qLmrWylqhywZE8ucv9rBqXwbTRvc69wOM993zWdXPvv5n5g5BZ+bALmfm4PAzc0jDLpSjqsyaNYuf/OQn3/ldQkICS5YsYdasWUydOpXf//73DXruxqr32Tiq+m9V/Y2q/tqaQc0G9wghMiTArsZsalX99Oerr76aefPmkZeXB0Bqairp6ekcO3aMoKAgZsyYwcMPP8zmzZu/89jmUuc3BBHJpebjAQRQVe3cLFW1UiLCZQMi+TIxDVe54uvT7s8QN2epfvrztddey/Tp05kwYQIAnTp14p133mH//v088sgj+Pj44O/vz8svvwzA/fffz7XXXkuPHj2abaOiONsY2rbRo0drfHy8R+b16bZjPDh/Cx/+/HuM6h3qkXma+tu9ezeDBw/2dhkeU9PrFZEEVR1d0/3tBP4mdumFEfiIHcZsWidrCE0sNLgDF/fqaocxm1bJGkIzuGxAJNtTsjmZX+LtUkwN2sNqMjTudVpDaAaTB0ahCmvsoiktTmBgIFlZWW2+Kah7OPjAwMAGPc7GZWgGw3t2ISy4Ayv3ZnDTiHOdxW08KTo6mpSUFJr7GhktQWBgINHR0Q16jDWEZuDjI0zsH8HqfRmUlys+tvuxxfD39z9jYBRzJltlaCaTB0aSlV/CzmONuvSkMV5hDaGZTOofiQh8vSfd26UYU2/WEJpJeKcARvUOZdmuNG+XYky9tfiGICJ9RWSuiCysNi1YRN4UkVdF5C5v1leXa4Z2J/F4DkeymveiFsY0lWZtCE00BsNBVb33rMm3AgtVdSbw/SYuu8lcPbQ7AEt3nfByJcbUT3N/Q3iDphuDobpoqq7S7Gqm2s9b7/AghvTobA3BtBrN2hC0icZgqEEKTlOAFr7ac81F3Uk4cop0u2iKaQW88cfUoDEYRCRcRGYDI91jQwB8CPxARF4GPq3lcR4bqKUuU4d2Q9X2NpjWwRsHJjVoDAZVzQJ+eta0fM4x2KyqzgHmgHP6c8PLbBoDu4XQvXMgq5My+I+xvb1VhjH14o1vCK16DIaGEhEmDYhgbVKmjdlgWjxvNIRWPwZDQ00aEElOURnbUuyoRdOyNfdux/nAOmCgiKSIyL2qWgZUjMGwG1igrWwMhoaquGjKartGgmnhmnUbgqreWcv0JbSjy7h3DerA8OiurNqXwa+vGuDtcoypVYveZdeW2EVTTGtgDcFDrhgcRbntfjQtnDUEDxnWswvdOwfyZaIdtWhaLmsIHiIiXDWkG6v2ZVBY0mKPtjbtnDUED5o6tBtFpeWs3Z/p7VKMqZE1BA8aFxtOSIAfy+xkJ9NCWUPwoA5+PkwZFMXy3WmU2lGLpgWyhuBhN424gFMFpXaQkmmRrCF42KQBkYQG+bNoS6q3SzHmO6wheJi/rw83DL+ALxPTyC0q9XY5xpzBGoIX3DyyJ8Vl5Sy1C7CaFsYagheM6t2VPuFB/DshxdulGHMGawheICJMG92LdQezOJSZ7+1yjKlkDcFLbo+LxtdHeH/T0XPf2RgPsYbgJVGdA7l8UBQLE45SUmbHJJiWocU3hFoGahksIrNFZKGI/Myb9Z2PO8f2IjOvhK9228ZF0zK0yoFaVHW3qv4UmAaMbvrKPeOyAVH07NqRt9Yd9nYpxgCtd6AWROT7wFrgq+Yrv3n5+ggzxvdh3cEs9p7I9XY5xrTagVpQ1U9U9XtAjWM7tpRxGc7lP8b0IsDPhzfXHfJ2Kca0zoFaRGSyiLwgIq9Qy7UZVXWOqo5W1dGRkZFNWH7TCg3uwE0jLmDR5lS7vJrxutY6UMtKYGWTVuVF917al4+2HOPHr2/k7XvH0aWjv7dLMu2UDdTSAgzsHsJLd41i9/Ecfvz6RopKW98VlY6fLuR0wfmdm6GqrEnK4OjJgiaqyjNKysopL/fa4GBNygZqaSGuHNKNf9w5ki1HsvmfTxo+TMXRkwWoev5DmVNUylOf7mLin1dwx5x1FJc1rJntS8vln18nsTAhhZlvxXP33I3c9doG8orL6nycq1z5MjGNgpK679fcVu5NZ+QfljHq6S954L3NnGrlq302UEsLcs1FPfjFlH68v+kozy9PYv3BLLILzv0BW7z9GBP/bwUvrthf73ntSDnNx1tTKSgp40BGHg9/sI1nl+4lrQGjVKsqv3h3M29+e4jJAyPZcyKX55cn1fvxH8Qf5cZ/rOXZZft4+INtrN2fyT2XxJByqoAnP679I5GVV8wP521g5lvx/L6O+9Wk1FXOx1tTmbc2mW/3Z+I6j//Zv9h5nJlvxdMnPJirBnfji50neObzPY1+vpbABmppYX595QC2p5zmb8v3AeAjENcnlEevGcSYmLDv3H/XsdM8/ME2fATmrD7I3RNi2J+ex9qkTH42uR8d/Jyefzgrn6c/242PQEGJizVJznUdQwL9KCp14efjQ1GZi9mrDvDI1QO5f1JfVOHbA1ksTDhKVOdAfnlFf4IDqj4yS3edYE1SJk/eOIR7Lonl0YXbmL3qAHF9QrlicLc6X+c3+zN5ZOF2vtcvnGdvv5iiUhedO/oT0SmAkEB/XvgqiQn9wrktLrryMW+tO8T8jUc5kJEHwMT+ESxMSOHmET25tH/Ed+aRlVfMfy/awaUXRnD98Av4aEsqc9cmk5pdWHmfGeN78/TNw9hwMIvnv0rihxP6cPXQ7ojUtKmrSqmrnMc/3MGQHp15+75xdA70JzS4A3NWH+SOsb0Y1Tu0zsc3hzJXOeVK5XveGOKNr5meNnr0aI2Pj/d2GfXmKlcOZuRxIqeITckn+XBLKuk5xfzl9uHcNKLnGfe78rlVFJW6+NOtw/jx65u48eIL+Hp3GvklLi4fFMXf7hhB4rEcHnhvMyWucrp1DqSwxMX0cb0Z2bsrCxNSCO7gx0NX9KewxMWfPt/N5ztPcHGvrqSeKiAzr4TOgX7kFpcRHdqR2TPiGHpBFwpLXFz53CpCAv1Y/OCl+Pn6kFtUym0vr2NvWi53ju3Fb68fckYDqe6/FmxjWeIJ4n97JQF+vmf8rsxVzt1zN5Jw+BTvzhzHmJgwPtqSyq/+tZURvboyJiaUW0dFExsRzHXPryGvuIwhF3TmVEEp2QUlTOgbzh9vGcbDH2z7zoVo4vqE8vPJ/RgW3YXnlu1jQfxRFv7sezw0fwup2YWowqDuIdwwvAfdu3QkLaeI7p0DGdQjBF8fISy4A1Ehgazcm86PX9/Eaz8czZVDnOaXV1zGFX9dSUigPw9d0Z8pAyMJCfzuBuKvdqexdNcJ/njLMPx9fSguc9HB1+ecTag25eXKZzuO89yX+5gxvg/3Xhpb5/1FJEFVazygzxpCK5BdUML9byewMfkkP5zQh1nXDqZjB1++3pPGf74Rzz+nj+SG4Rdw/1vxLEtMo2fXjkwf15tnl+2l4u2NDu3Im/85ln6Rneqcl6ry2ppk5m88wrDoLlwxuBtTh3RjR+ppHpq/BQEWPzSRpz9L5MPNqcyfOZ4J/cIrH19U6uJvy/fx6uqD9I8K4ZW744iJCD5jHsVlLkY/vZypQ7rz12kX1/qab33pWzLyihkbE8aapExG9u7K2/eOO+N/wC1HTvHEop34+ghdg/zx9/Xh6z3pfK9fON8eyOLByy9kbGwYa/dnct1FPbi4V9cz5nHZX1ZSUlZOYamL+TPHc/RkAfM3HWHLkewa6wrw8+HTBy/l1dUH+WLnCeJ/d2ZDW7E3nd/8ayunCkrpEx7Epw9eSudqTWF/eh7f/+daCkpc/M+NQ7huWA++/89vuHZYd568cWid783pglL+9PluZozvw0U9uxB/6CRz1yaz5Ug2J3KKGNgthP++fjCXDah7N7s1hFbeEMD5I/rz53uZ900yF0Z1YsFPJvBfC7ay81gO3z5+Of6+PhzIyOOZz/fw39cNJjYimFX7Mthy5BS9w4KYMjCK0OAO51XD9pRsbnt5HV2D/EnPLeZXV/bnV1fWPFblmqQMHpy/hZKycm6Li+aeS2KJdTeG5Ylp3PdWPK/fM4YpA2s9GJVDmfn875LdHDlZQFhwB16cPqper+EvS/fw4ooDxEYE8/kvJxLo71vrfeetTeYPixO5a1xv/njLsMrp6blFFBS7iOocQOqpQval5VGuyhOLdjAsugs7U3O4YlAUz90x4jvP6SpXVuxJ5yfvJHDtRd35x50jERHyi8u45aVvyMwrITYimKS0XAb16MzG5JN08PVh7WNTiOocWGutD3+wjYUJKYQG+fObqQN5enEiIYH+jO8bxlVDunHD8Avw9Tn3twxrCG2gIVRYvS+D+96KZ3D3ELannubBKRfym6kDPTb/t9cf5ncf7eSHE/rw1PeH1vk1N+VUAc99uY/F246jKPdN7Mv9E/vyh8WJfL0nnU1PXHle67u1UVXe23iEMTFhDOgWUud9y1zOlasuHxRFxw61N44KFQ0EYO6PRte5reTFFfv5y9K9PDDlQu4Y04sH3tvMjtTTvHHPWLp1DuS6F9bgKld+PrkfL686wM8u68ej1wyqfHypq5z/XbKbzLwSLo7uwtOf7eb2uGhW7ssgI7eYQd1DePe+cYR3CqjnknFYQ2hDDQHgw80p/GbBNnx9hLWPTaFHl44enf+BjDxiw4Pxqcf/RuD8b/t/X+xlYbUrRN0xuhd/vm14c5XYbEpd5Vz999Vk5BbXuP2jOle58sv3t7B4+3HAWd345/RRXOXe5jB3bTKZecU8evVAfvHeZtYkZTJ/5niCOviSXVjKX5ft5Zv9WXQK8COvuIz+UZ1Y/NClHM4qYP7GIzx0ef9GfeuzhtDGGgI4/1MVlrr4xZQLvV1KvW1PyWb9wSyOnCzgnktiz7k9o6VKzsznZH4JcX3qtydh69Fs3t94hNtH96r1MduOZnPTi9+cMc3fV/jTrcO59qLuLNlxnDExYd/ZHtMY1hDaYEMwbc/Wo9kczy6kqMxF144duDCqE73Cgpp8PnU1BG+cy2CMqcGIXl0ZUW0viDe0+CsmGWM8xxqCMaaSNQRjTCVrCMaYSu1iL4OIZAC1Xck0Asj0YDnnYvXUzeqpW33q6aOqNR7f3C4aQl1EJL62XTDeYPXUzeqp2/nWY6sMxphK1hCMMZWsIcAcbxdwFqunblZP3c6rnna/DcEYU8W+IRhjKrXrhtCQMSabaf69RGSFiOwWkV0i8kv39DAR+VJEktz/euwCfSLiKyJbRGSxO8eKyAZ3Lf9yXynbU7V0dQ/ou8e9jCZ4edn82v0+7RSR+SIS6MnlU9NYqbUtD3G84P5sbxeRUfWZR7ttCLWNMenhMsqA/1LVwcB44BfuGh4HvlLV/jhjV3qyWf0S52rYFf4M/M1dyyng3hof1TyeB75Q1UHAxe66vLJsRKQn8BAwWlUvAnxxhhDw5PJ5g7PGSqX25XEt0N99ux94uV5zUNV2eQMmAEur5VnALC/X9DFwFbAX6OGe1gPY66H5R7s/VJcDi3FG2coE/GpaZs1cS2cgGfd2rmrTvbVsKoYgDMM5S3gxcLWnlw8QA+w81/IAXgHurOl+dd3a7TcEGjjGZHMTkRhgJLAB6KaqxwHc/9Z+4cGm9XfgUaDcncOBbHXG0gDPLqO+QAbwunsV5jURCcZLy0ZVU4FngSPAceA0kID3lk+F2pZHoz7f7bkhNGiMyeYkIp2AfwO/UtUcL9VwA5CuqgnVJ9dwV08tIz9gFPCyqo4E8vHsqtMZ3OvmNwGxwAVAMM7X8rO1lN12jXrv2nNDaBFjTIqIP04zeFdVP3RPThORHu7f9wDSPVDKJcD3ReQQ8D7OasPfga4iUnEhHU8uoxQgRVU3uPNCnAbhjWUDcCWQrKoZqloKfAh8D+8tnwq1LY9Gfb7bc0Pw+hiT4lyyeC6wW1Wfq/arT4AfuX/+Ec62hWalqrNUNVpVY3CWxdeqehewArjNk7W46zkBHBWRiktKXwEk4oVl45AuMzwAAAKgSURBVHYEGC8iQe73raIeryyfampbHp8AP3TvbRgPnK5YtaiTJzbItNQbcB2wDzgAPOGF+V+K8zVuO7DVfbsOZ939KyDJ/W+Yh+uaDCx2/9wX2AjsBz4AAjxYxwgg3r18PgJCvblsgKeAPcBO4G0gwJPLB5iPs/2iFOcbwL21LQ+cVYYX3Z/tHTh7R845DztS0RhTqT2vMhhjzmINwRhTyRqCMaaSNQRjTCVrCMaYStYQjNeJyOSKsyuNd1lDMMZUsoZg6k1EZojIRhHZKiKvuK+dkCcifxWRzSLylYhEuu87QkTWu8/FX1TtPP0LRWS5iGxzP6af++k7Vbv2wbvuowERkWdEJNH9PM966aW3G9YQTL2IyGDgDuASVR0BuIC7cE7y2ayqo4BVwJPuh7wFPKaqw3GOlKuY/i7woqpejHMuQMXhtCOBX+Fcm6IvcImIhAG3AEPdz/N0875KYw3B1NcVQBywSUS2unNfnFOl/+W+zzvApSLSBeiqqqvc098EJolICNBTVRcBqGqRqha477NRVVNUtRznEO4YIAcoAl4TkVuBivuaZmINwdSXAG+q6gj3baCq/k8N96vrWPiaTsmtUFztZxfORUfKgLE4Z4PeDHzRwJpNA1lDMPX1FXCbiERB5bX8+uB8hirO9psOrFXV08ApEZnonn43sEqdaz2kiMjN7ucIEJGg2mbovk5EF1VdgrM6MaI5Xpip4nfuuxgDqpooIr8FlomID84Zd7/AuXDJUBFJwLmK0B3uh/wImO3+gz8I3OOefjfwioj8wf0ct9cx2xDgYxEJxPl28esmflnmLHa2ozkvIpKnqp28XYdpGrbKYIypZN8QjDGV7BuCMaaSNQRjTCVrCMaYStYQjDGVrCEYYypZQzDGVPr/X6dOF9jAg84AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "L2 norm of w: 1.3071598\n"
     ]
    }
   ],
   "source": [
    "fit_and_plot_tf2(0, lr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 205,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-12T05:05:57.400638Z",
     "start_time": "2020-04-12T05:05:49.695602Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPoAAAC1CAYAAAB24uKEAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO2dd3gVVd7HPyedVAghEAhVOqy0wIpiQUVBZe0F1/K6Kq5lxY68dl13sexrWRQsoCjFAiJFegcpIQkBQhJIaElIQkJIDzfl3t/7x9w0SUICyb3ce8/nefJk5syZme+cO985Z04bJSJoNBrnxs3eAjQaTcujja7RuADa6BqNC6CNrtG4ANroGo0LoI2u0bgAHvYWcC4opcYD4wMCAh7t3bu3veVoNHYnOjr6pIi0q2+7cuR29IiICImKirK3DI3G7iilokUkor7tuuiu0bgA2ugajQugja7RuAAOWRnXWN5csp/wNq145PIe9paiaWHKy8tJS0vDZDLZW0qL4uPjQ3h4OJ6enk3azyGNXlnr3rNnzwbj/Z58kp6h/jxyuW10aexHWloaAQEBdOvWDaWUveW0CCJCTk4OaWlpdO/evUn7OmTRXUSWisjEoKCgBuN5e7pRWmGxkSqNPTGZTLRt29ZpTQ6glKJt27bnVGpxSKM3Fh8Pd0orzPaWobERzmzySs71Gp3a6N6ebpSW6xxd0/Lk5eXx+eefN3m/G264gby8vBZQVBvnNrqHOyado2tsQH1GN5sbvv+WL19O69atW0pWFQ5ZGddYvD10jq6xDS+//DKHDh1i8ODBeHp64u/vT1hYGLGxscTHx3PLLbeQmpqKyWRi0qRJTJw4EYBu3boRFRVFUVER48aNY9SoUWzbto1OnTqxePFiWrVq1Sz6nNroPp7uujLOBXlr6X7i0wua9Zj9OwbyxvgB9W6fOnUqcXFxxMbGsnHjRm688Ubi4uKqasdnzZpFcHAwp0+fZvjw4dx+++20bdu21jGSkpKYP38+X331FXfddRcLFy7kvvvuaxb9Dll0V0qNV0p9mZ+f32A8bw83TOW66K6xPSNGjKjVBPbpp58yaNAgLrnkElJTU0lKSjpjn+7duzN48GAAhg0bxtGjR5tNj0Pm6CKyFFgaERHxaEPxvD1085or0lDOayv8/Pyqljdu3MjatWvZvn07vr6+XHXVVXU2kXl7e1ctu7u7c/r06WbT45A5emMxiu46R9e0PAEBARQWFta5LT8/nzZt2uDr60tiYiI7duywsToHzdEbi1F0tyAiLtHGqrEfbdu25bLLLmPgwIG0atWK9u3bV20bO3YsM2bM4OKLL6ZPnz5ccsklNtfn3Eb3dAegzGzB28Pdzmo0zs68efPqDPf29mbFihV1bqt8Dw8JCSEuLq4q/IUXXmhWbU5ddPf2MC5Pv6drXB3nNro1R9dt6RpXx7mNbs3RdRObxtVxSKM3pR0ddNFdo3FIozd2mKpPZdFdN7FpXByHNHpjqS666xxd49o4udF1jq6xDec6TBXg448/pqSkpJkV1capje7jqd/RNbbhQje6c3eY8dDNaxrbUHOY6pgxYwgNDeWnn36itLSUW2+9lbfeeovi4mLuuusu0tLSMJvNvPbaa5w4cYL09HRGjx5NSEgIGzZsaBF9Tp2je1fl6Lro7nJ8cyPsnmssm8uN9T0/GutlJcZ63EJj3ZRvrMcvMdaLc4z1A9bebIUnznq6qVOnctFFFxEbG8uYMWNISkoiMjKS2NhYoqOj2bx5MytXrqRjx47s2bOHuLg4xo4dy9NPP03Hjh3ZsGFDi5kcnNzoPrrDjMYOrF69mtWrVzNkyBCGDh1KYmIiSUlJ/OlPf2Lt2rVMnjyZLVu2cLZWo+bEyYvuOkd3WR76rXrZ3bP2updv7XWfoNrrfm1rrwdUD1BpDCLClClTeOyxx87YFh0dzfLly5kyZQrXXXcdr7/+epOOfa44dY7eKi+JPipFN69pWpyaw1Svv/56Zs2aRVFREQDHjx8nKyuL9PR0fH19ue+++3jhhReIiYk5Y9+WwiFz9MZ+wKHVikm84lHK3oprbSNM47LUHKY6btw47r33XkaOHAmAv78/c+bMITk5mRdffBE3Nzc8PT2ZPn06ABMnTmTcuHGEhYW12Hu6U382WVIjuXF6FNdcdQ3PX9fHhso0tiYhIYF+/frZW4ZNqOtaz/bZZIfM0RuL6jyCI+6n9KAWjcvj1O/oFKRzrcduysrL7a1Eo7Erzm30xN/4r0zFw5RrbyUajV1x6qI7fW/i8fVmfC3NMwm+5sLGFeYGPNc6NefO0QPDOOTdj2Kzcz/PNMZ3w3Nycs7ZCI5A5WeTfXx8mryvczugopRREk3J6W7AMHur0bQg4eHhpKWlkZ2dbW8pLYqPjw/h4eFN3s+5jW4u4/WCt5jHI8Bt9lajaUE8PT1rfRlFUxvnLrp7+fN66Kes9NIdZjSujXMbXSnS/fqTY/G3txKNxq44t9GB/uX7GXx6p71laDR2xSHf0Rvb1x1gTN7P+JpSgOb98oVG40g4ZI7e2FlgAX4Lf4an3F+zgSqN5sLFIXP0pmDy7chxPWecxsVxyBy9KbQ3Z3CbeaUxXZBG46I4vdE7lSbzpvssJPeovaVoNHbD6Y2e2e4yRpg+ozTYNcYqazR14fRGd/f2J4s2lFqce7CDRtMQTm90b3fFve7rsBzaaG8pGo3dcHqj+3h58IzHQjwTfrG3FI3Gbji90b093BhbOpXMK6baW4pGYzdcwuinCMRUYW8lGo39cHqj+3i6018dJWT7u1BWbG85Go1dcHqje3u40U1lEho/C/LT7C1Ho7ELzm90T3dWWyLYfMdeaKfndte4Jk5vdB9PNyrw4LRZt6NrXBenN3rlN9I7J3wFUbPsrEajsQ8uYHTjEkMzN8HRrXZWo9HYB5cx+sqIr+EOnaNrXJNGGV0pNUkpFagMZiqlYpRS17W0uAb0jFdKfZmff/ahp75expD74jI9Jl3jujQ2R/+biBQA1wHtgIcAu3U1a8oMMz6ebnh7uOFxMhEWPgInk22gUKO5sGis0SurrG8AvhGRPTXCLmiUUgT7eXHKZDHe0QvT7S1Jo7E5jZ1KKloptRroDkxRSgUADlMWDvbz4mB5ADyXAE7+bS6Npi4aa/SHgcHAYREpUUoFYxTfHYJgPy9ySsq1yTUuS2OL7iOBAyKSp5S6D3gVcJhJ2IL9vMgtKYOMvTD9MkiLtrckjcamNNbo04ESpdQg4CXgGPBdi6lqZtr4enGqqAwCwqBVG7DooWwa16KxRfcKERGl1M3AJyIyUyn1YEsKa07a+nlRWFpBmU9bvP5nmb3laDQ2p7E5eqFSagpwP/CbUsod8Gw5Wc1LGz8vAKP4DlBaaPxpNC5CY41+N1CK0Z6eCXQCPmgxVc1MW6vRTxWXQckp+GggbP/czqo0GtvRKKNbzT0XCFJK3QSYRMRx3tFrGt03GC6bBL3G2FmVRmM7GtsF9i4gErgTuAvYqZS6oyWFNSeVOXpOsbXofvlz0GmoHRVpNLalsZVxrwDDRSQLQCnVDlgLLGgpYc1J1Tt6pdEBirIh+hsY9Sy4O0x1g0ZzTjT2Hd2t0uRWcpqwr91p3coTpWrk6ADpMbDx35Cyw37CNBob0dgcfaVSahUw37p+N7C8ZSQ1Px7ubgS18qydo/e6Dv4RA8Hda0e2mMHN3bYCbUVhpnF9QZ3sreTcMOWDz9kHMmnOpLGVcS8CXwIXA4OAL0VkcksKa26C/byMyrhKlKo2eeouEIET++HdDrBjxpkHyD5oG6EA5vKmhdfHofUwaxyczjPW50+AuXeAxQLJ62D2X6DAOsjHVAAVpXUfR6Tur9EWZUHuUeN49SFSvZz4G5zOrQ7/436nc2vHr2Tje7DiZaO15ND6Gte3AebcXt1UemJ/w1oaonK/03mw+jXY18BbqcV8bueoj7wUOLiqdpgIbHofjv7eLKdodPFbRBaKyHMi8qyILGqWs9uQYN8/GL2SI5th5rUQtxDaD4CRT0GXP9eOs2M6TL8UMvcZN3fhCSPcYqndHl8ZDsYPlX+8ev3722DR3+sWd2g97Jpp7BO/BKZFGMcqyICY7404W/4PvrsFyk8b6+byamOKQFmJVUMmnDpsLFeUQmFGdS54zevwl2ng5mboPp1r9BYEWDoJPh1SfZyaLHzY6Dr8xxv86BaYNsL4D3A8BnZ+Wa3pl8dg5cvGekE6/HAvHN5kPDS+vQmiZ0H0bDi23Yiz9k2YOab2eSxmOLgCyothwC0Q1Ll6W+sukJUIKTvhyBbjN0pYYvxOu7424pSfhlWvGCMXT+yHWWONhxqA2dpDsqwEvr7GMLeHD6RGQlZ89fkz9kBFGWz5j3GuGZfDtmkQ+RWUm4zjJq2pvu6aD2RTQfV5Dq6ufrDW3P7ZnyH7QO1wpcBcBr/+vXr/86BBoyulCpVSBXX8FSqlCs777DbkjBy9ki6Xwk0fQ7/xxvq1b0DHIcYNsuZ1KM6BQRNgzFsQfBF8cQWstBZmljxl5CgVZZAeC58MMh4YAL8+Ad+Mrc6huoyEsMHG8ulcY7+8FGM9LRrWv2Ost+0JIX2MH3rnDFjxknFztO4CrTuDm4fxAPmwF+z90dh/8VPwwwTDQNMvNW5CgD7j4B/R1YN5LhoN4cOM5f43w6MbqrddfBfc+B/w8oX4xfDp0GrTD7gVrn3TeKWxWOD3T4wbtM+NcNNHEHaxES9hCax7G0qLjOP6BhtdjsF4oEzcZKx7B4JfCLh7w/ZpxnUC9BhtaK58dUpeC0UnDJ3jPoC//BdCehnTdu+eA20vgkfXQa9roeulcN27xjXu+cHQEfUNnIiHA8uNh8m+n42KV09fiP4Wpo80fufyEvAJNLR5+sDfVhoPRTBy1S+uMH6byK9gz3xo3x+KMmH5C8Zv8NU1kJ9qpM2Ch4xwMNJvamfYOd14cP90v/HbHI+BBQ8bcXwCjesaeJux/6LHjfMAjHoO/roA3Bv7ht0AIuKwf8OGDZPGMnnBHon455pGx5fMOJF3QkVi59cO37dAJPugsRz3i8iWj0QsFhFTociKl0WKc4xthzaI7J4nUlF+5rELMkW+uFJk0wfVYaXFImZz7XimQpGTyWfub7GILH9J5NgOYz36O5HNHxrhB9eIpMc2/jrr4tBGkTl3ihRmnbnt2HaRt4JFDqw6c1txjoipoGnnMhVWp1lNsg6IvBEk8vunZ2777UWRN1uL5B+v+5jFOYb2D3qJLHikWpPFIlKQYSwf3iTy80MiRSert9VFeqzIvoXGcs30sFhEju82lvf+LJKXaiyve6f6nhAR2f65SPI6Y/nYdpHyUpFN74t8MuTM31tE5LtbRDa+V7eWBgCipAGvKKnrnchBiIiIkKioqEbFfX9lIl9uPkzSu+NQjR2uWpAOgR3PQ6GTkhkH7l7QrnfLncNigexEo1QQ0KH2NlO+Uapp37/hY5ScMl5fAsNaTue5IAKmvOrSTjOglIoWkYj6tjdDmcAxCPbzosIiFJgqCGrVyHZzbfK66TCw5c/h5la/kX2CGlf77hvcvJqaC6Wa1eSNwWHaws+XNr41usFqNC6Gyxg92L/S6PU0IWk0TozLGD00wBuAEwXa6BrXw2WMHhbUCoDMfJOdlWg0tsdljN7G1xMvDzcyC7TRNa6HyxhdKUWHQB8ydI6ucUEumOY1pZQf8DlQBmwUkbnNfY4OQT6c0EbXuCAtmqMrpWYppbKUUnF/CB+rlDqglEpWSlk7Q3MbsEBEHgX+0hJ6woJ8yCg43RKH1mguaFq66P4tMLZmgHViyc+AcUB/YIJSqj8QDqRaozXz8CADI0cvxWJx3N6AGs250KJGF5HNwKk/BI8AkkXksIiUAT8ANwNpGGZvUJdSaqJSKkopFZWdnd0kPR0CfSgzWzhVojvNaFwLe1TGdaI65wbD4J2AX4DblVLTgaX17SwiX4pIhIhEtGvXrkknDgvyAXQTm8b1sEdlXF0jSkREimnh77l1qNGWPrCTnqlE4zrYI0dPA2rMHkA4YJNvGVfm6Bm6LV3jYtjD6LuAXkqp7kopL+AeYIktThzi7427myIzX9e8a1yLlm5emw9sB/oopdKUUg+LSAXwFLAKSAB+EpH9TTzueKXUl/n5Tfugq7ubIjTAm8x83d9d41q06Du6iEyoJ3w55zGLrIgsBZZGREQ82tR9OwT5kKnb0jUuhst0ga0kLEh3g9W4Hi5n9PaBPmTmm3DkKbQ0mqbickbv1LoVJWVmPdOMxqVwSKOfa2UcQN8OgQAkZurvo2tcB4c0uogsFZGJQUFN7/TSNywAgIQMh5qWXqM5LxzS6OdDiL83If7eOkfXuBQuZ3SAfmEBJGbqHF3jOrio0QM5eKKICvM5fpBPo3EwXNLofTsEUFZh4cjJYntL0WhsgkMa/Xxq3aG65j1Bv6drXASHNPr51LoD9Az1x8NNkahr3jUugkMa/Xzx8nCjZ6i/bmLTuAwuaXSAAR2D2J2ah1nPH6dxAVzW6Ff2aUdeSTmxqbn2lqLRtDiua/Re7XB3U6xPzLK3FI2mxXFZowf5ejKsSxvWJzZtJlmNxhFxSKOfb/NaJaP7hpKQUUCGnlpK4+Q4pNHPt3mtkqv7hgKwQefqGifHIY3eXPRu70+XYF8WRKfqiSg0To1LG10pxcQrehCTkseWpJP2lqPRtBgubXSAOyPC6Rjkw0drD3KquIzoY7q5rbE4QinovZWJ/O+ifYChd3HscYpLK+ysyva4vNG9Pdx58uqe7E7JY+g7a7h9+jY2Hbyw3tmjj+WyZE/tb1ycLCrl1V/3nfF5qRX7Mth+KKfFNZWbLdzw6VY+XHXgrHFLyir49/IEThadfZrtlJwSvtp8uNZDpNxsYcov+4g73vTK15VxmSyMTsNUbiYmJZdJP8QyPzKlVpxXf93Hv5cnNPnYjcVU3iLfDG0SLm90gDuHdeb+S7ry7LW9aePryc9RqbW2bzqYzUsL9nDnjG1VN9vGA1m8tGBPvV9mzSkqZU9qHsfzzq9Gv9BUzuNzonnh5z0U1ciJvtpymDk7Unj5l71VpjCVm3n2p1jum7mTxbHHz+u8pnJzgzn2lqRsEjIKmLYhmZVxGQ0e67e9GXyx+TD/spopM99U73wAH6w+wLvLE2q9Sm08kM38yBTeWRZf7zlEhEW701gbf4KsQlPVNRzLKaa0wkL0sdyqSteaD8LNB7OZsyOFeTtTKDdbMFuE77cfJb+kvNbxi0orapUEGtujct7OFIa8vaZJD98fIlOYs+NYs5Y87PHttQsOLw833rllIAC5JWXMi0whv6ScIF9P0vNO8/C3u/D38aC4tIIfd6UysFMQM7ceYUvSSS7v1Y5r+oXyP7N2MaZ/ex69ogeLdqfx/E97sIhx7CVPXUZYYCue+XE3Ywd24O7hXarOvTj2OD9HpdEtxJfRfUK5um8oSlV/nu6jNUlkFRo54YbELMYP6khxaQXzd6YQ4u/FxgPZLI5N55YhnYg6moup3EKn1q145sdY/L09uKZfe/ak5vHJuiTKKiwM7dKaSdf2xt1NsS/NeGh1atOKYD+vqnMezzvNLZ/9zuW9QvjPnYNQSnGiwMTRk8WEBHhzUTt/FsemE9TKk25tfXnx570M6BhE52BfwDDYE3Nj6NrWlzfGD2BlXCYAv8QcZ1TPEP61PBFTuZnIV67B16v6FswuLK16aHy99QhX9DY+ovmT9cG788gpdh09xfBuwQCcKDBRaKqgZ6g/M7ce4Z+/GQ8SdzfFmmevwFRuodKPW5NPstlaUtt55BQVZmPbm0v34+XhRmFpBbtT8sgrKeO1xfuJPpbLx/cMASAtt4TbPt9Gpzat+OXxS9mbls/dX25n5oPDuaxnCMWlFczefpRZW4/g5+3Bk1f15LahnXB3U8zcepjT5WYe/S6KHyZewsBOQZgtwkdrDhJ55BQFpnKm3NCPK63XeiCzkCmL9iEC761IZNpfh1ZtOx8cMkdvrnb0urhjWDhlFRaW7DWKyl9vOYIAy/4xiit7t2PDgSyKSyvYedj4GvRHaw/yr+UJRB49xdSViWw8kMWbS+IZ1Lk10/86lEAfT56at5uJ30ex4UA2/7sojt+TjdyqqLSCt5bGk5BRwOLYdB6eHcVt07fx9++juem/W7h/5k5mbz/KhBGdCfH3YuV+wzALY9IoMFUw475hDOnSmreW7qfQVM7mpGy83N1Y+o9R9Gznz7u/JVTl8rtTcsk7Xcan65N55sdYnpoXw/hpWxk/bSvD/rmGx+dEE30sF1O5mSfnxnCyqJRfYo7zc3QabyyO48//WsfdX+5g7Meb2XboJKv3n+CGP4Ux7d6hWER49dc4RASLRXj+pz2sT8xi9raj7E/PZ0vSSe4Z3pnQAG+e+2kPJWUVFJVWsGKfcT1bkrI5UWDip6hUys3CbUM6sflgNgcyC8kqNLE+MYsHRnYl2M+LaeuTASMHf+ibXVz/8WZeWbSPqSsSGdO/PTMfjMBsEbYfziEpyxiGHOLvxdI96exPL2BAx0CKSiuISy/gu+1HOZxdzAd3XIy7m2LzwWwWW1+Rfo1NZ9PBbLIKTDwwK5Kc4jJ2p+Sxan+mNV0tzNh0CIBJP8Ty/soD9AsLJMDHg5cW7uWtpfFEH8vlUHYxz17bm6BWnjwyO4oCUzm/xKQxbUMyZWYLZWYLf/t2F3N3HkNE+GTdQfy8PJj9txF0bN2K53+KJacRrzxnwyFz9PP5UsvZGNAxkL4dApi/M4UreoUwPzKFmwd1JLyNL6P7hrI2IYvvth+jzGzhgZFd+W77MQ5nF3PnsHA2HszmoW934enmxgd3DKJnqD/+Ph7cPzOSZODdWwcye9tRnpwXw7xHLmHDgSxOFZex6IlL+VOnIH6OTmPGpkPkl5QTHuxLbnEZ/cMCmTy2L0opft19nJyiUmZuPcLgzq2J6BbMm+MHcPNnv/NDZCqbD2YT0a0NwX5evHh9HyZ+H82DsyI5nF3MVw9EMKZ/ez7fmMz7Kw/g6a54bkxv+nYIYHdqHnO2H2NFXCZeHm6UVVj474QhfL/9GC8t2AvAAyO7MrpPKK8tjuN/vtlFWYWFWwZ3pHOwL89d14d3lsUzPzKV7Ydz+G1fBo9d2YNvth7l8TkxlJkt3D4snKv6tOPD1Qf59J4hPDE3mp+iUmnt68nDs6MI8PHAy92NkT3a8tpN/Vkel8HUFQl0C/HDbBEeGNmN9oE+fLDqANuST1JSZiY+o4CLw4OYuzOFzsGt+PCOQQS28iDE34voY7l0at0KdzfF3cM789kGw5STx/blgVmRrI0/wdydx7i8Vwg3D+7E99uPsWp/JimnSrhneGcij57iiTnRnC434+HuxpyH/8yrv+7jpQV7KTBV0C8skC1JJ/l++1HWJpzgxev78OTonogIby2N59ttR4k+louflzuPXN6dq/q049bPf+ftpfFsTTrJoM6tWfTEpRSXmXlqXgyvLIpjbfwJNhzI5umre3Jl73Z0CPRh/H+3MuWXfXxx/7BaJb2m4pBGb0mUUjx6eQ+e/3kPoz/ciEXgsSsvAuCqPkYHm2nrk/D39uCVG/uxJy2fQlM5b988kK3JJ3n0uyieHN2TnqH+AFzeqx3v327kGLcPC2dUzxDu+mI7t03/HQ83N67uG8qQLm0AmDCiCxNGdKlT17iBHZi3M4Wxn2zhVHEZb99svGoM6tyakT3aMn3TIU4VlzF5bF8AxvRvz5Aurdl55BSX9Ajm2n6G9ieu6km/sEDCgnyqJuC4bkAHHr/qItYnZLHjcA49Q/0ZP6gjQ7u2YdL83dwzogt3DAsH4DO/odwxYxsdg3yqitAPjuzKwug0/nfRPjzcFM+P6c1TV/ckr7icH6NSaRfgzbAubXBzU4wdGAbAnRGd+WDVAZKyiujd3p/Wvl5EHjnFg5d2pY2fF3+/8iI+XpsEB7IZ2qU1PUP9CQvqxsKYNCb9GEuIvzedg1ux8PFL2Xn4FF3b+hLk6wnAkC5t2J2SR3FpBV3b+nJ13/Z8tuEQ7QO9ubxXCD1D/Zmx6RAVFuGF6/oAcEXvdvzfmoNV2u4e3pmpKxIZ0T2Y8YM60rt9AM9f14cn5sbQo50fs/82nMvf28Bri/fTMciHh0d1r7p/Xh7Xly1J2cRnFDBhRGf8vD0Y1Lk1D13WnZlbjwDw6YQhKKXw9/Zg5oPDmb4xmY/WJhHg48HDo3oA0KdDAC+N7cM/f0tg26EcLusZcu43tog47N+wYcOkpdh/PF+emBMtby3ZXyv8+o82SdfJy+Sx76JERKS4tFxKSiuqtqeeKhaLxdLgsbMKTHL3F9ukx5TfZF9aXqP0lFWY5eI3V0nvV5bLmv2ZtbZtSDwhXScvk66Tl0nc8erjRR3NkdEfbKgV1hz8npQtkUdyaoXFp+fL0/NjJD49vyosOatQur28TF5dtO+MY2TknZbuLy+THlN+k72peWI2WyQxo6BWnBMFp2VRTJokZxVWhSVk5EufV5dL18nLZO6OY3Xqm74xWbpOXibD3lkjE7/bJeUVZhny9mp5ZdFeERF5ddE+6Tp5mTwye1fVPrtTcqXr5GUy6r119f5+ZrNF/rMqUWJTckVE5KWf90jXyctkQVTqGXH3pObKFe+vl4SM6vQoLi2Xqz/cIE/Pj6nz+PHp+bInNfeMc65PPHHWewqIkga8Ynezns9fSxq9Pv69PEG6Tl4mP0amnNdxKswWycg73aR9oo6ekv3H888It1gscv1Hm2TYO2vEbG74hrA1kUdyJLe4tM5t/1mVKN/+fqTJx1y2J10enb1LTOUVdW7feTin6sH34apEERFJzyuR4tJyERHZfDBLer+yvNZDqcJskdEfbpAZG5MbrSMz/7R8veWwVDQhzUvLzS3yG53N6EocoNNDfUREREhUVJRNz5mcVcQbS+KYNmEobWrUVNuboyeLKTRV8Kfw8+v/7wyYys0MfGMVFRbhk3sGc/PgTmfEKauw4OXhkHXRdaKUihaRiPq263f0JtIz1J+5j1xibxln0C3Ez94SLhh8PN0Z0DGQPWn59AoNqDOOM5m8MbjW1WpchuHdgvFyd6NHO/0ABJ2ja5yUf6TqUCIAAAXKSURBVFzTi78M7oiPp7u9pVwQ6Bxd45QEtfLk4vDW9pZxweCQRm/JnnEajTPikEaXZpphRqNxFRzS6BqNpmloo2s0LoBDd5hRSmUDx+rZHAJcSPNDaT0No/XUT2O0dBWResezOrTRG0IpFdVQTyFbo/U0jNZTP82hRRfdNRoXQBtdo3EBnNnoX9pbwB/QehpG66mf89bitO/oGo2mGmfO0TUajRWnNLpSaqxS6oBSKlkp9bKNz91ZKbVBKZWglNqvlJpkDQ9WSq1RSiVZ/7exsS53pdRupdQy63p3pdROq54flVI2G1yvlGqtlFqglEq0ptNIe6aPUupZ628Vp5Sar5TysWX6KKVmKaWylFJxNcLqTA9l8Kn13t6rlBramHM4ndGVUu7AZ8A4oD8wQSnV34YSKoDnRaQfcAnwpPX8LwPrRKQXsM66bksmATW/UvAe8JFVTy7wsA21fAKsFJG+wCCrLrukj1KqE/A0ECEiAwF34B5smz7fAmP/EFZfeowDeln/JgLTG3WGhqafccQ/YCSwqsb6FGCKHfUsBsYAB4Awa1gYcMCGGsKtN8vVwDJAYXTA8KgrzVpYSyBwBGv9UI1wu6QP0AlIBYIxhm0vA663dfoA3YC4s6UH8AUwoa54Df05XY5O9Q9XSZo1zOYopboBQ4CdQHsRyQCw/g+1oZSPgZcAi3W9LZAnIpWfArFlGvUAsoFvrK8SXyul/LBT+ojIceBDIAXIAPKBaOyXPpXUlx7ndH87o9Hrmvza5k0LSil/YCHwjIjU/f0h2+i4CcgSkeiawXVEtVUaeQBDgekiMgQoxvavMVVY331vBroDHQE/jOLxH7lQmqfO6bdzRqOnAZ1rrIcD6fXEbRGUUp4YJp8rIr9Yg08opcKs28OALBvJuQz4i1LqKPADRvH9Y6C1UqpyhiFbplEakCYiO63rCzCMb6/0uRY4IiLZIlIO/AJciv3Sp5L60uOc7m9nNPouoJe11tQLo2Jlia1OrozPacwEEkTk/2psWgI8aF1+EOPdvcURkSkiEi4i3TDSYr2I/BXYANxhBz2ZQKpSqo816BogHjulD0aR/RKllK/1t6vUY5f0qUF96bEEeMBa+34JkF9ZxG8QW1R42PoPuAE4CBwCXrHxuUdhFKX2ArHWvxsw3ovXAUnW/8F2SJergGXW5R5AJJAM/Ax421DHYCDKmka/Am3smT7AW0AiEAd8D3jbMn2A+Rj1A+UYOfbD9aUHRtH9M+u9vQ+jteCs59A94zQaF8AZi+4ajeYPaKNrNC6ANrpG4wJoo2s0LoA2ukbjAmija1oEpdRVlSPlNPZHG12jcQG00V0cpdR9SqlIpVSsUuoL67j1IqXUf5RSMUqpdUqpdta4g5VSO6zjoBfVGCPdUym1Vim1x7rPRdbD+9cYdz7X2vMMpdRUpVS89Tgf2unSXQptdBdGKdUPuBu4TEQGA2bgrxgDO2JEZCiwCXjDust3wGQRuRijV1Zl+FzgMxEZhNFPvLJL5hDgGYx5AXoAlymlgoFbgQHW4/yzZa9SA9rors41wDBgl1Iq1rreA2M464/WOHOAUUqpIKC1iGyyhs8GrlBKBQCdRGQRgIiYRKTEGidSRNJExILRFbgbUACYgK+VUrcBlXE1LYg2umujgNkiMtj610dE3qwjXkP9pOsaNllJaY1lM8ZEDhXACIzRfbcAK5uoWXMOaKO7NuuAO5RSoVA1T1lXjPuicuTWvcBWEckHcpVSl1vD7wc2iTHWPk0pdYv1GN5KKd/6Tmgdpx8kIssxivWDW+LCNLXxOHsUjbMiIvFKqVeB1UopN4zRU09iTAYxQCkVjTHjyt3WXR4EZliNfBh4yBp+P/CFUupt6zHubOC0AcBipZQPRmng2Wa+LE0d6NFrmjNQShWJiL+9dWiaD11012hcAJ2jazQugM7RNRoXQBtdo3EBtNE1GhdAG12jcQG00TUaF0AbXaNxAf4fxJ1BZbGvKS0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "L2 norm of w: 0.34986514\n"
     ]
    }
   ],
   "source": [
    "fit_and_plot_tf2(3, lr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 228,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-12T05:27:22.970575Z",
     "start_time": "2020-04-12T05:27:22.962596Z"
    }
   },
   "outputs": [],
   "source": [
    "def fit_and_plot_tf2(wd, lr=1e-3):\n",
    "    net = models.Sequential()\n",
    "    net.add(layers.Dense(1))\n",
    "    net.build(input_shape=(1, 200))\n",
    "    w, b = net.trainable_variables\n",
    "    optimizer = optimizers.SGD(learning_rate=lr)\n",
    "    train_ls, test_ls = [], []\n",
    "    loss = tf.keras.losses.MeanSquaredError()\n",
    "    for _ in range(num_epochs):\n",
    "        for X, y in train_iter:\n",
    "            with tf.GradientTape() as tape:\n",
    "                l = loss(net(X), y) / 2 + wd * l2_penalty(w) / batch_size / 2\n",
    "            grads = tape.gradient(l, net.trainable_variables)\n",
    "            optimizer.apply_gradients(zip(grads, net.trainable_variables))\n",
    "        train_ls.append(loss(net(train_features),\n",
    "                             train_labels).numpy())\n",
    "        test_ls.append(loss(net(test_features),\n",
    "                            test_labels).numpy())\n",
    "    semilogy(range(1, num_epochs + 1), train_ls, 'epochs', 'loss',\n",
    "                 range(1, num_epochs + 1), test_ls, ['train', 'test'])\n",
    "    # print(test_ls)\n",
    "    print('L2 norm of w:', tf.norm(w).numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 229,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-12T05:27:31.586574Z",
     "start_time": "2020-04-12T05:27:23.217914Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQQAAAC1CAYAAAC56YFuAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAe3UlEQVR4nO3deXhUhbn48e87k8m+QQKYEEKAogUEWQKCViu9dbetK5tW23rV9unvtrf3aitPa3vt3ey9vfeqbbUudauKWrR1Q0UoriwhwQBhkTWQECAkhOz7vL8/zmSBZoVkziR5P88zT/KeOcs7JzNvzjlzznlFVTHGGACP2wkYY0KHFQRjTCsrCMaYVlYQjDGtrCAYY1pZQTDGtApzO4FgSE5O1oyMDLfTMMZ1OTk5Jao6orPnh0RByMjIIDs72+00jHGdiBzo6nnbZTDGtLKCYIxpZQXBGNPKCgJAYy38+Xuw/yMnrquATx+Co9uduLYM1v4Wjn3uxNWlTlyyx4mrip34+D4nrjzixGX5Tlxe6MTlhU584qATVxQ58fH9Tlx51IlL9zpxdYkTH9vlxDXHnbh4hxPXlTvx0W1OXF/pxIe3OHFDjRMXfebETfVOXJjjxM1NTlyQ5cQt17UcWAfrHmlbP/mfwvrft8X7PoSsJ9rivWtg4x/a4j2rIOeZtnjXStj0x7b483cg98W2eMdbsPnltnj7G7DlT21x3mvOo8XW5bDtL23xlldgx5ttce4y2LmiLf7sedj1Xluc8yzsXtUWZz8Ne1a3xRufhH0ftMUbHof9H7fF6x+FA2vb4rW/hYMbnN/9zU5cGDhm1dzoxIdynLix1okPb3bihmonPrLViesqnPh033tnaMAVBBGJEZFnReQJEbm5T2Zae8IpBi0f0LoT8P7PoWiTE9cch5U/bfsjVh114qN5TlxR5MTFO534RIETt/zRyvKduOWPVrrHiU8cdOKSXU5cESgYR7c5ceURJz6yxYlbCkTRZ05cW+bEhRuduKUgFGxw4sZAQcj/1IlbCsL+D53YHygIe//qxC32vA/v39cW73oHVt/fFu98G/76b23x9tfhgwfa4rzX4KNft8VbX4FP/rct3rzMKbgtPnse1v22Lc55Bja0K0DZTzkf0hZZj0PO023x+kdg03Nt8drfQO4LbfEnD8Lml9rij38Necvb4g9/BdvbFZg1/3FygVn9S6eItXj/FycXmJU/hb2BgqJ+J963xombG5w4/xMnbqx14gPrnLi+yokLspy47oQTd/beqy4OvPcCBeTU994ZklC42lFEngKuAYpV9dx2w68AHgK8wJOq+oCIfBM4oapvisjLqrqwu/lnZmZqr75l8PudD1NYBHh9TtVvqAZfVM/i5iZnel80eMN6EDc6b5TwGPB4u4+bGqCpDsJjwePpQVzvPCLiQKT7uLHOeSNHxjvro9u41ikuEXEdxw01zgclIraHcbXzMzymZ3F9FYgHwqN7GFeCJ8z5e/UkrqsAbzj4IruPVZ3pwyKcR3ex3w8NVRAWCWHhHcS9fO91Q0RyVDWz0+dDpCBcDFQBz7UUBBHxAruAS4FCYCOwGPgG8I6q5orIi6q6pLv597ogGDNIdVcQQmKXQVU/Ao6fMngOsEdV96lqA/ASTjEoBNIC45xx/n6/8i9vbOPJj/tmH8yYgSwkCkInRgMF7eLCwLDXgBtE5FHgzY4mBBCRO0UkW0Syjx071ulCPB5hT3EVz67Lx+93f2vJGDeFckGQDoapqlar6rdV9Xuq+kIH47SM+LiqZqpq5ogRnZ6pCcANs0ZTcLyWrPxTN1KMGVpCuSAUAmPaxWlAUX8s6PIpZxET7uXVnML+mL0xA0YoF4SNwEQRGSci4cAi4I3+WFB0eBhXTU1hxdbD1DQ09ccijBkQQqIgiMgyYB1wjogUisjtqtoE/D/gPWAH8IqqbuuvHG6clUZ1QzPv5h3pr0UYE/JC4mpHVV3cyfAVwIqOnutrszOGkz48mj9lF3L9zLTuJzBmEAqJLYRQ4PEIN81KY92+Ug6W1ridjjGusILQzg2z0hCB5TkF3Y9szCBkBaGd1MQoLpo4guU5hTTbOQlmCLKCcIoFmWkUldfx6Z4St1MxJuisIJzi0smjSIz28XK27TaYoccKwikiwrxcO3007287Sll1g9vpGBNUVhA6sHD2GBqa/fwl95DbqRgTVFYQOjApJZ5paQm8vLGAULg83JhgsYLQiQWZY9h5pJKth8rdTsWYoLGC0ImvT08l0ufh5Y12cNEMHVYQOhEf6eOqc1N4I7eI2oZmt9MxJiisIHRhwewxVNY3sWLrYbdTMSYorCB04fxxw8lIirZzEsyQYQWhCyLCTZljyNp/nP0l1W6nY0y/s4LQjRtnpeH1CK/YVoIZAqwgdGNUfCTzz3EueGpq9rudjjH9asAVBBGZJCK/F5HlIvK9YCxz4ex0jlXWs+bzzu/ebMxgENSCICJPiUixiOSdMvwKEflcRPaIyL1dzUNVd6jqd4EFQKcNJ/rS/HNGMCIugpc3HgzG4oxxTbC3EJ4Brmg/INCh6XfAlcBkYLGITBaRqSLy1imPkYFpvg58AqwmCMK8Hm6clcaaz49xtKIuGIs0xhVBLQi96dCkqltV9ZpTHsWB+byhqhcAfdPstQcWZI6h2a8st1u1m0EsFI4hdNahqUMicomIPCwij9HFDVh72rmpp8YlxzB3/HBeyS6wDk9m0AqFgtBhh6bORlbVD1T1B6p6l6r+rovxety5qacWzU7nQGkN6/eX9sn8jAk1oVAQgtah6Uxdce5ZxEeG8Ypd8GQGqVAoCEHr0HSmIn1erpsxmhV5RyivaXQ7HWP6XLC/dnS9Q9OZWjg7nYYmu5uSGZyC2rkpFDo0nanJqfFMHZ3AsqyD3DpvLCIdHQIxZmAKhV2GAWfhbOduSlsK7W5KZnCxgnAavj49lSif1y6LNoOOFYTTEB/p4+ppzt2UquutfbwZPKwgnKZFs8dQVd/E23Y3JTOIWEE4TbPGDmPCiBheyrILnszgYQXhNIkIi2ans+ngCXYdrXQ7HWP6hBWEM3D9zNH4vGK3ajeDhhWEM5AUG8Flk8/itU2F1DfZrdrNwGcF4QwtmjOGsppGVm476nYqxpyxoJ6pOBhdOCGZtGFRvLTxIF87L9XtdEw3GhsbKSwspK5ucN/oJjIykrS0NHw+X6+ms4JwhjweYWHmGP7n/V0cKK1mbFKM2ymZLhQWFhIXF0dGRsagPe1cVSktLaWwsJBx48b1alrbZegDN2WOwSPYwcUBoK6ujqSkpEFbDMD5BiwpKem0toKsIPSBsxIimX/OSP5kt2ofEAZzMWhxuq/RCkIfWTTHuVX7X3cWu52KMafNCkIfmX/OCEbFR/CS7TaYLpw4cYJHHnmk19NdddVVnDhxoh8yOpkVhD4S5vVw06wxfPB5MUUnat1Ox4SozgpCc3PX57GsWLGCxMTE/kqrlRWEPrRw9hj8ivWBNJ2699572bt3L9OnT2f27NnMnz+fJUuWMHXqVACuvfZaZs2axZQpU3j88cdbp8vIyKCkpIT8/HwmTZrEHXfcwZQpU7jsssuore27f0AD7mtHEfEA/wrEA9mq+qzLKbUaMzyaiyYm88rGAv7hKxPxegb/wauB7P43t7G9qKJP5zk5NZ5ffG1Kp88/8MAD5OXlkZubywcffMDVV19NXl5e69eDTz31FMOHD6e2tpbZs2dzww03kJSUdNI8du/ezbJly3jiiSdYsGABr776Krfcckuf5D/gWrkB38Dp29CIc8fmkLJodjpF5XV8tMv6QJruzZkz56RzBR5++GHOO+885s6dS0FBAbt37/6bacaNG8f06dMBmDVrFvn5+X2WT4+2EETkh8DTQCXwJDADuFdVV/Zyec8AvwWeazfvllZul+J8wDeKyBuAF/jPU6b/DnAOsE5VHxOR5QSpnVtPXTp5FEkx4SzLOsj8L450Ox3Tha7+kwdLTEzbiWwffPABq1atYt26dURHR3PJJZd0eC5BRERE6+9er7dPdxl6uoXwHVWtAC4DRgDfBh7o7cL6qJVbIVAWmLbTIzF93bmpp8LDnD6Qq3cWU2x9IM0p4uLiqKzs+HL58vJyhg0bRnR0NDt37mT9+vVBzq7nBaFlZ/gq4GlV3UzHHZdOR69auQGvAZeLyG+AjzobqT86N/XUwtlOH8g/WR9Ic4qkpCQuvPBCzj33XO65556Tnrviiitoampi2rRp3HfffcydOzfo+fX0oGKOiKwExgFLRSQO6KtT8nrbyq0GuL2Plt0vxo+IZe744by08SDf+/IEPHZw0bTz4osvdjg8IiKCd955p8PnWo4TJCcnk5fXdgju7rvv7tPcerqFcDtwLzA78IH04ew29IUB08qtNxbPSafgeC1r91ofSDNw9LQgzAM+V9UTInIL8DOgr5oSDJhWbr1x+ZSzSIz2sWyj3XPRDBw9LQiPAjUich7wY+AA7b4p6KnB0MqtpyJ9Xm6YmcbKbUcoqap3Ox1jeqSnBaFJVRXnHICHVPUhIK63C1PVxaqaoqo+VU1T1T8Ehq9Q1bNVdYKq/ntv5xuqFs8ZQ2Oz8qodXDQDRE8LQqWILAW+CbwdOHegd7diGYK+MDKO2RnDeGljAU49NSa09bQgLATqcc5HOILzteB/91tWg8ii2ensL6lm/b5TT78wJvT0qCAEisALQIKIXAPUqWqvjyEMRVdPSyE+Moxl1tDFcPqXPwM8+OCD1NTU9HFGJ+tRQRCRBUAWcBOwANggIjf2Z2KDRaTPy/Uz03g37wjHqxvcTse4LNQLQk9PTPopzjkIxQAiMgJYBSzvr8QGk0VzxvDM2nxe21TI31803u10jIvaX/586aWXMnLkSF555RXq6+u57rrruP/++6murmbBggUUFhbS3NzMfffdx9GjRykqKmL+/PkkJyezZs2afsmvp8cQPC3FIKC0F9MOeV88K56Z6Ym8mHXQDi6Gmqevhs9ecH5vbnTizS87cUONE+e96sR15U68PXCaTHWpE38eOLuwsvveHA888AATJkwgNzeXSy+9lN27d5OVlUVubi45OTl89NFHvPvuu6SmprJ582by8vK44oor+MEPfkBqaipr1qzpt2IAPf9Qvysi74nIt0TkW8DbwIp+y2oQWjwnnX3HqsnabwcXjWPlypWsXLmSGTNmMHPmTHbu3Mnu3buZOnUqq1at4ic/+Qkff/wxCQkJQcupR7sMqnqPiNwAXIhz7cHjqvrnfs1skLlmWiq/fGs7y7IOcv74pO4nMMHx7bfbfvf6To7Do0+OIxNOjmOSTo7jRvVq0arK0qVLueuuu/7muZycHFasWMHSpUu57LLL+PnPf96reZ+uHm/2q+qrqvpPqvojKwa9FxXu5boZo1mRd4QyO7g4ZLW//Pnyyy/nqaeeoqqqCoBDhw5RXFxMUVER0dHR3HLLLdx9991s2rTpb6btL11uIYhIJR1feSiAqmp8v2Q1SC2ek85z6w7wqh1cHLLaX/585ZVXsmTJEubNmwdAbGwszz//PHv27OGee+7B4/Hg8/l49NFHAbjzzju58sorSUlJ6bfjCDIUDnJlZmZqdna222kAcN0jn1JR28iqf/rykGgYEmp27NjBpEmT3E4jKDp6rSKSo6qZnU1j3xQE2eI56ew9Vs3G/LLuRzYmyKwgBNk101KIiwjjxQ0H3E7FmL9hBSHIosPDuG6mHVx001DYTT7d12gFwQWL56TT0OTn1U12WXSwRUZGUlpaOqiLQks7+MjIyF5PO+AatQwGk1LimT4mkWVZB7n9S+Ps4GIQpaWlUVhYSDDvxO2GyMhI0tLSej3dgCsIIjIZ+Bec06dXq+qAvJ5iyfnp/Hj5FrL2H7cTlYLI5/Od1BjFnGwgdm66EviNqn4PuLXfku1nLQcX7bJoE0qCfQzhGeCK9gPadW66EpgMLBaRySIyVUTeOuUxEvgjsEhE/hsYsP9a7eCiCUVBLQh90bkp8Pg+zm3hS4KZf1+zg4sm1ITCtwy96twkIhki8jjOXZ87vY2bW63cemNSSuCy6A12WbQJDaFQEHrbuSlfVe9U1ZtV9ZMuxnOtlVtvLDl/LPtKqlm3zxq6GPeFQkEYlJ2beuqawD0XX9xgBxeN+0KhIAzKzk09FenzcsOsNN7bdoRSa+hiXBbsrx2HTOem3lg8J91p6GIHF43Lgnpikqou7mT4CobwLdnOHhVH5thhvJRVwB0XjbczF41rQmGXwQCL5qSzr6SaDXbPReMiKwgh4uqpKcRZQxfjMisIISIq3Mv1M0bzjjV0MS6yghBClpw/1jlz0bpFG5dYQQgh55zlHFx8Mesgfr+duWiCzwpCiLl5rtMt2s5cNG6wghBirjw3hcRoH39cZ/dcNMFnBSHERPq8LMwcw/s7jnK4vNbtdMwQYwUhBN18/lj8qizLKuh+ZGP6kBWEEJSeFM0lZ49gWdZBGpr8bqdjhhArCCHq1nkZHKus571tR9xOxQwhVhBC1JfPHkH68GieXZvvdipmCLGCEKI8HuHWeWPJPlBG3qFyt9MxQ4QVhBB2U+YYonxenluX73YqZoiwghDCEqJ8XDdzNK/nFtn1DSYorCCEuG9fkEF9k5/n19uJSqb/WUEIcRNHxfGVL47k2bX51DU2u52OGeRCviCIyHgR+YOILG837FoReUJEXheRy9zMLxjuuGg8pdUN/PmzQ26nYga5fi0IfdG6LdDA5fZThv1FVe8AvgUs7PPEQ8zc8cOZOjqBJz/eZ/0bTL/q7y2EZzjz1m1d+VlgXoOaiHDbBRnsPVbNxvwyt9Mxg1i/FoS+aN3W0XzF8SvgHVXd1Mk4Id+5qTeumnoWsRFhvLTRbrFm+o8bxxB627otSUR+D8wQkaWBwf8AfBW4UUS+29F0A6VzU09Fh4fx9emprNh6mIq6RrfTMYOUGwWht63bSlX1u6o6QVX/MzDsYVWdFRj++37LNMQszBxDXaOfN3KHTGMrE2RuFIQh3brtTExLS2BKajwPrtrFgdJqt9Mxg5AbBWFIt247EyLCQ4tm0ORXbnsqy1q/mT7X3187Wuu2PvaFkbH84bZMDpfXcdNj6yg4XuN2SmYQkaHwvXZmZqZmZ2e7nUaf2ph/nL9/Nhuf18O/X3cul00eZS3gTLdEJEdVMzt7PuTPVDQdm50xnOXfnUditI+7/pjD9Y+u5Xdr9rCl8ISdvGROm20hDHBNzX6WbSzgubX57C6uAmBsUjS3zcvg1nljCfNazTdtuttCsIIwiJRU1fPXHcUs31RI1v7jnDs6nrsunsDFZ48gIcrndnomBFhBYOgUhBaqyttbD/Ovb23naEU9IjAyLoIpqQn8x3VTOSsh0u0UjUusIDD0CkKLZr+SW1DGp3tKKThewzt5R4iNCOOJWzOZmpbQb8vNL6nmwPEaLpiQhC+wy6Kq7DhcSVxkGInRPj47eIK1e0v5dE8JxZV1ACRGhZMcFw6A1+Nh7PBoUhOjiPR5qG/yU1bTwIjYCM4eFUdGUgwpiZGt8+9vzX7F6xH8fmXHkQpSE6IYFhMelGX3JSsIDN2CcKrtRRV855mNHKmoY9bYYcRGhLGvpIpbzh/LnRePR0Qorqzjna1HaGz2c9XUFFISIqmqb8KvUNfYTHZ+GUcq6hg/IoZDZbW8veUwIjB+RAzxkT4Onajlzc1F+BXOio/khlmjOX9cEk9/up81n598TUmYR5iZPoxxyTEAlNU0UFJVj4jQ0OQnv6Sayvqm1vF9XqGx+eT3a7jXQ0piJBdPHMG45BjCvM60x6sbWL+vlPzSGlISIkkfHs34ETFcOCGZueOT8Hg6/0am2a/sPeYcjyk6UctDq3ezpbCc8ckxlNU0UlJVT0SYh2umpeIRqKxrYtbYYZw7OgGvRxibFM2o+K63wtbvK+W/3t1JamIUS85PZ3JKPD6vh+2HKzhR00hybDiFZbVsOlhGYlQ440fEEOYRGpqd1xbmEcYlxzJzbCLR4WEAHC6vJSUhqsvlWkHACkJ7pVX1vJJdyOu5zr0VYiPCyD5QxjXTUlo/RC19ZkXAK0JTF41nvzAylpiIMPYfq6K2sRmf18PN56czM91pWrt2bynNfiXK5+WHX51IYpSP0uoGpqUlMDN9GDERYZ3OW1WpbWymrtFPpM9DdHgYJVX17D5aRcHxGo5U1FHT0Mye4irW7i2hpqHtBjIegWlpiUxKieNIeR0HSms4cLyGZr+SNiyKORnDGRkfyY7DFRwur2VEXASRYV4q65rYcbjipEKUmhDJVVNTyC+tJio8jIsmJvPZwTJezy0iNiKMqHAvB0prTlr2lyaOYHxyDM1+pbq+qTU3f+A1fby7hNGJUVTWNVJR17asU7VsHXX2MR0ZF8EPvzqRrP3HeXNzEa9//0tdbv1ZQcAKQlf8fuWBd3fy+Ef7GJ8cwzXTUvjaeamEh3l4a8thquubGBYdjojzH33amETSh0ez71g1sRFhTEqJO+n8B1U9KS6vaWRj/nEmpcYzOrHr/15norHZT019M41+Pz6vhyifl/Cwk3cn6hqbeW/bEV7PLWJ7UQXFlXWcPSqOMcOjOVZZT0OTn9iIML4wKpbMscMID/PgFeErk0YSEeb9m2W2f62Hy2vZd6wavypZ+4/zem4R5bWNeARiIsKIDvciCCLgEWHehCT++bKzEYQPdx2jsKyGusZmJqXEkxwbQUlVPcmxEUxJjaexWTlwvBpVCA/zMDw6nPomP9sPl/N/7+9m66Fyonxebrsgg7suHt/lrowVBKwg9ERVfRMx4d4hdXJTy3GBgazZr2zYX8rZo+JIjo3odvzuCkLn22tmSIntYtN9sBroxQCc13DBhOQ+m5+dtWKMaWUFwRjTygqCMaaVFQRjTKsh8S2DiBwDOmt9lAyUBDGd7lg+XbN8OteTXMaqaqc3GR0SBaErIpLd1dcwwWb5dM3y6Vxf5GK7DMaYVlYQjDGtrCDA424ncArLp2uWT+fOOJchfwzBGNPGthCMMa2GdEHoTRfqflj2GBFZIyI7RGSbiPwwMHy4iLwvIrsDP4cFOS+viHwmIm8F4nEisiGQz8uBXhrByiVRRJaLyM7Aeprn5voRkR8F/lZ5IrJMRCKDuX466qbe2foI9D99OPDe3iIiM3uyjCFbEDrrQh3EFJqAf1bVScBc4PuB5d8LrFbVicDqQBxMP8Tpl9HiV8D/BfIpA24PYi4PAe+q6heB8wJ5ubJ+RGQ08AMgU1XPBbw4TYaCuX6e4ZRu6nS+Pq4EJgYedwKP9mgJqjokH8A84L128VJgqYv5vA5cCnwOpASGpQCfBzGHtMCb6ivAWzh9OEuAsI7WWT/nEg/sJ3Ccq91wV9YPbU2Kh+NcJfwWcHmw1w+QAeR1tz6Ax4DFHY3X1WPIbiHQyy7U/UlEMoAZwAZglKoeBgj8HBnEVB4Efgz4A3EScEKdblsQ3HU0HjgGPB3YhXlSRGJwaf2o6iHg18BB4DBQDuTg3vpp0dn6OK3391AuCL3qQt1vSYjEAq8C/6iqFcFefrs8rgGKVTWn/eAORg3WOgoDZgKPquoMoJrg7z61CuybfwMYB6QCMTib5acKla/tTutvN5QLgutdqEXEh1MMXlDV1wKDj4pISuD5FKA4SOlcCHxdRPKBl3B2Gx4EEkWk5e4pwVxHhUChqm4IxMtxCoRb6+erwH5VPaaqjcBrwAW4t35adLY+Tuv9PZQLgqtdqMW5V9kfgB2q+r/tnnoDuC3w+204xxb6naouVdU0Vc3AWRd/VdWbgTXAjS7kcwQoEJFzAoP+DtiOS+sHZ1dhrohEB/52Lfm4sn7a6Wx9vAHcGvi2YS5Q3rJr0aVgHJAJ1QdwFbAL2Av8NMjL/hLOJtwWIDfwuApnv301sDvwc7gL6+US4K3A7+OBLGAP8CcgIoh5TAeyA+voL8AwN9cPcD+wE8gD/ghEBHP9AMtwjl804mwB3N7Z+sDZZfhd4L29FefbkW6XYWcqGmNaDeVdBmPMKawgGGNaWUEwxrSygmCMaWUFwRjTygqCcZWIXNJyZaVxnxUEY0wrKwimR0TkFhHJEpFcEXkscN+EKhH5HxHZJCKrRWREYNzpIrI+cB3+n9tdo/8FEVklIpsD00wIzD623X0PXgicCYiIPCAi2wPz+bVLL31IsYJguiUik4CFwIWqOh1oBm7GucBnk6rOBD4EfhGY5DngJ6o6DecsuZbhLwC/U9XzcK4DaDmVdgbwjzj3pRgPXCgiw4HrgCmB+fxb/75KA1YQTM/8HTAL2CgiuYF4PM5l0i8Hxnke+JKIJACJqvphYPizwMUiEgeMVtU/A6hqnarWBMbJUtVCVfXjnMKdAVQAdcCTInI90DKu6UdWEExPCPCsqk4PPM5R1X/pYLyuzoPvqvd6fbvfm3FuONIEzMG5GvRa4N1e5mxOgxUE0xOrgRtFZCS03sdvLM77p+VKvyXAJ6paDpSJyEWB4d8EPlTnXg+FInJtYB4RIhLd2QID94lIUNUVOLsT0/vjhZmThXU/ihnqVHW7iPwMWCkiHpyr7b6Pc9OSKSKSg3MHoYWBSW4Dfh/4wO8Dvh0Y/k3gMRH5ZWAeN3Wx2DjgdRGJxNm6+FEfvyzTAbva0Zw2EalS1Vi38zB9x3YZjDGtbAvBGNPKthCMMa2sIBhjWllBMMa0soJgjGllBcEY08oKgjGm1f8HHDEKVgllciAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "L2 norm of w: 1.3485094\n"
     ]
    }
   ],
   "source": [
    "fit_and_plot_tf2(0, lr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 224,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-12T05:18:30.036425Z",
     "start_time": "2020-04-12T05:18:22.121394Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAC1CAYAAACQyynAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAXVklEQVR4nO3deXxU9bnH8c+TyQaZAGaBsqhBQVxQWaKFqq3eFgWsVavFqtjNl7S3WrW3daGtbe1ypb1d1KviVkUrpXBFBRQRQRBXMNFgw6JhbQJIEpCQACHbc/84Z5gQE5iEzJxwzvN+vfLK/M6ZOXnmZL5zfr8z55wRVcUYEzxJXhdgjPGGhd+YgLLwGxNQFn5jAsrCb0xAWfiNCahkrwuIh5ycHM3Ly/O6DGM8V1hYWKmqua3N82X48/LyKCgo8LoMYzwnIpvbmmfdfmMCylfhF5FLROTRqqoqr0sxpsvzVfhVdZ6qTurZs6fXpRjT5fkq/DFrrId9n3pdhTGeCl74Gxtg6hdg4S+8rsQYT/lyb/8hhZIh/3rIHuR1JcZ4KnjhBxj1A68rMMZzwev2R9TtgbcfgJ0bvK7EGE8EN/y1u2HpPbDqea8rMcYTwez2A/ToCz98B3od53UlxngiuFt+iAZ/9zao3+dtLcYkmK/C36Ej/GrK4aHPw+t/jF9hxnRBvgp/h47wC/eGL90BwyfGrzBjuqDgjvmbG31j9HZTEyT56j3RmFbZqzxCFV78Mcz/ideVGJMQtuWPEIG0HqBNzhuBiNcVGRNXFv7mvvJrC70JDOv2NxcJfsXH8Pevw55Kb+sxJo4s/K3ZXw0VH8HurV5XYkzcWLe/NQNGws3vQ3Ka025qhKSQtzUZ08lsy9+WSPALp8HTl8L+Gk/LMaazWfgPJzUMaZkQSvW6EmM6lXX7D+f0K2HoFc7OwNoqqFznDAuMOcrZlj8WkU8BXv0VTBsPNRXe1mNMJ+jyW34RyQAeAuqApao63bNixtwNJ42FsPsFKOVroffJnpVjzJHwZMsvIk+ISLmIFLeYPlZEPhKRdSJypzv568CzqnoD8LWEF9tcek8YMta5vW0lPDTK2SFozFHIq27/NGBs8wkiEgIeBMYBpwJXi8ipwACg1L1bYwJrPLTck2HcH+EU9/1o01vw2u+dYwSMOQp4En5VXQbsbDH5bGCdqm5Q1Trgn8ClQBnOGwB0pX0UyWnw+UnQPctpb3oTCv4GyelOe9ULsOIx7+prqanRuWw5OOcuVG2BfbvceU1OT6b6E6fd2AAblsKn7te8NdTB6jmwY73TrtsLHzzjHAkJzhve8kedYRA434nw1n1QvsZp76mEpX+Izq/+BBb/NtreVQoL74LKEqe9cwMs+Fn071WWwPzbo9dbLF8DL/0kWt+2D52Tsqq2OO0thTD35ujzKV0Bc38U3Vez6S2YcyPsdV+CG16HF250Lu0GsG4xvPBD53kCfPwKPP+fznoAWPsSPP8DZ72B879+4YfRdf2vZ2HOTdH2ypkw79Zo+4NnnPojCp+Cl++Itt97HF75ebS9/BFn/US88yAsupsj1XXCBP2JbuHBCX1/4DngChGZCsxr68EiMklECkSkoKLCgx1y598BtxZDKMVpr3rO+SdHzJwIM66Ott/4M7w7NdoueMJ50USseOzg9tIpBy9vzk3OiyLi6cucZUbcPwIW/yba/n1feK1Z+6+nOS8icE5meuSL8P7TTruxzjm2IXJ9w/q9MOtbTggA6mqc8Gx83WnXVsHLt0HZCqe9dye8+ksnlOCG/7+hfLXTrimHN/8CO9dH28sfgV1umGsqnFoiR1jWbIcPZ0YPt67+BIqfg307o+0185w6AKq3O7VGjs3YvRVKXnWeB0D1Nli/BBr2R+dvWOp8mQvA7i2wcRk0ue2qMufNXd2O565S2PwWoO78Utj8dnTd7vq384ZzoL0Zyt6Ltj/dBFvej7Z3boCtRc3aG50344gd62D7qmi7siS6Lo+EqnryA+QBxc3a3wAeb9a+Dvjfjix75MiR2iXs3xO9/fYDqm/eG21Pn6A66zvR9tRzVadfFW0/OEp15nXR9qMXqM69Jdp++nLV134fbc++QXX5o9H2gp+pFj8XbS/7s+r6JdF24VOqW953bjc1qa6ep1rxsdNubFTd9JbqrjK33aC67V+qNZXR9s5NqrW7o+2aStW6fdHH769RbaiPLr+xwfltEgoo0DZyIs78xBORPOBFVR3qtkcDv1bVi9z2ZABVvae9y87Pz9ej7iu6VZ0fu5CI6UQiUqiq+a3N60qvtPeAwSIyUERSgW8Cc9uzgKP6W3pFLPgmobz6qG8G8A4wRETKROR6VW0AbgJeAdYAs1R11aGW05Lat/QaEzNPDvJR1avbmD4fmJ/gcowJJF/1M4/qbr8xCear8Fu335jY+Sr8xpjY+Sr81u03Jna+Cr91+42Jna/Cb4yJnYXfmIDyVfhtzG9M7HwVfhvzGxM7X4XfGBM7C78xAeWr8NuY35jY+Sr8NuY3Jna+Cr8xJnYWfmMCqst/aYcxR6K+vp6ysjJqa2u9LiWu0tPTGTBgACkpKTE/xsJvfK2srIzMzEzy8vKQyNeu+YyqsmPHDsrKyhg4cGDMj/NVt9/29puWamtryc7O9m3wAUSE7OzsdvdufBV+29tvWuPn4Ed05Dn6KvzGmNgFLvylO/fyoxkfsLJ0l9elmADYtWsXDz30ULsfN378eHbtiu9rNHDhr61vZN7KrWzeudfrUkwAtBX+xsZDf+fs/Pnz6dWrV7zKAgK4tz8z3fkopKa2weNKTBDceeedrF+/nmHDhpGSkkI4HKZv374UFRWxevVqLrvsMkpLS6mtreWWW25h0qRJAOTl5VFQUEBNTQ3jxo3j3HPP5e2336Z///7MmTOHbt26HXFtgQt/ON15ytW19R5XYhLt7nmrWL11d6cu89R+PfjVJae1OX/KlCkUFxdTVFTE0qVLufjiiykuLj7wkdwTTzxBVlYW+/bt46yzzuKKK64gOzv7oGWUlJQwY8YMHnvsMSZMmMDs2bOZOHHiEdfuq25/LB/1ZaSGEIGa/bblN4l39tlnH/RZ/P3338+ZZ57JqFGjKC0tpaSk5DOPGThwIMOGDQNg5MiRbNq0qVNqiWnLLyK3AE8C1cDjwHDgTlVd2ClVdBJVnQfMy8/Pv6Gt+4gI4bRkqq3bHziH2kInSkZGxoHbS5cuZdGiRbzzzjt0796d888/v9XP6tPS0g7cDoVC7Nu3r1NqiXXL/z1V3Q1cCOQC3wWmdEoFHuiRnmLhNwmRmZlJdXV1q/Oqqqo45phj6N69O2vXruXdd99NaG2xjvkjRxCMB55U1ZVyFB854Wz5bcxv4i87O5tzzjmHoUOH0q1bN/r06XNg3tixY3n44Yc544wzGDJkCKNGjUpobbGGv1BEFgIDgckikgk0xa+s+MpMT7Yxv0mYf/zjH61OT0tL4+WXX251XmRcn5OTQ3Fx8YHpP/3pTzutrljDfz0wDNigqntFJAun639UCqcns6OmzusyjPFUrGP+0cBHqrpLRCYCvwCO2rNnMtNTbMtvAi/W8E8F9orImcDtwGbg6bhVFWc25jcm9vA3qKoClwL3qep9QGb8yoqvHun2UZ8xsY75q0VkMnAdcJ6IhIDYLxnSxYTTktnf0ERdQxOpyb46zsmYmMX6yr8K2I/zef8nQH/gf+JWVQfFejGPTPcQXxv3myCLKfxu4KcDPUXkq0Ctqna5MX+sF/MIuyf32LjfxFtHT+kFuPfee9m7N35nn8YUfhGZAKwAvgFMAJaLyJVxqyrOMg+c3GNbfhNfXTn8sY75fw6cparlACKSCywCno1XYfGUmWbhN4nR/JTeMWPG0Lt3b2bNmsX+/fu5/PLLufvuu9mzZw8TJkygrKyMxsZG7rrrLrZv387WrVu54IILyMnJYcmSJZ1eW6xj/qRI8F072vHYLufAOf025g+eJy+GD6Y7txvrnfbKmU67bq/TLp7ttGurnPbquU57zw6n/ZF7VF719sP+uSlTpnDiiSdSVFTEmDFjKCkpYcWKFRQVFVFYWMiyZctYsGAB/fr1Y+XKlRQXFzN27Fhuvvlm+vXrx5IlS+ISfIg9wAtE5BUR+Y6IfAd4CZgfl4oSIHxgh5+N+U3iLFy4kIULFzJ8+HBGjBjB2rVrKSkp4fTTT2fRokXccccdvPHGGyTqArQxdftV9TYRuQI4B+ckn0dV9fm4VhZHNuYPsO++FL0dSjm4ndr94HZ6z4PbGdkHtzOjJ+nEQlWZPHky3//+9z8zr7CwkPnz5zN58mQuvPBCfvnLX7Zr2R0R85V8VHU2MDuOtSRM2Mb8JkGan9J70UUXcdddd3HttdcSDofZsmULKSkpNDQ0kJWVxcSJEwmHw0ybNu2gx+bk5MSltkOGX0SqAW1tFqCq2iMuVcVZekqI1FCShd/EXfNTeseNG8c111zD6NGjAQiHwzzzzDOsW7eO2267jaSkJFJSUpg6dSoAkyZNYty4cfTt2zcu435xjtr1l/z8fC0oKDjkfUb89lXGn/45fnfZ6QmqynhhzZo1nHLKKV6XkRCtPVcRKVTV/Nbuf9TusT9SmXZ8vwm4wIY/nJZsl+82gRbY8NuWPzj8OLRtqSPPMbDhD6elUG0H+fheeno6O3bs8PUbQOQrutPT09v1uC7/pR0icgLO4cU9VbXTzifokZ7MWjuxx/cGDBhAWVkZFRUVXpcSV+np6QwYMKBdj4lr+EXkCeCrQLmqDm02fSxwHxACHlfVNi8DrqobgOtFpFPPIwjbRTwDISUl5aAvyTBR8d7yTwMeoNklv9wLgTwIjAHKgPdEZC7OG8E9LR7/vRbnFHSayJhfVQPx/e3GtBTX8KvqMhHJazH5bGCdu0VHRP4JXKqq9+D0EjpERCYBkwCOO+64w94/nJZCY5NSW99Et9RQR/+sMUctL3b49QdKm7XL3GmtEpFsEXkYGO5eSqxVqvqoquaran5ubu5hi8i0L+w0AefFDr/W+tht7opV1R3ADzq7iOyMVAB27Kmjd4/27SU1xg+82PKXAcc2aw8AtnbGgmO9hh9Abqbz5YcV1fs7408bc9TxIvzvAYNFZKCIpALfBOZ2xoJjvYYfWPiNiWv4RWQG8A4wRETKROR6VW0AbgJeAdYAs1R1VTzraE1O2A1/jYXfBFO89/Zf3cb0+cThSkAicglwyaBBgw5734y0ZDJSQ5TvtvCbYPLV4b3t6faD0/W3Lb8JKl+Fv71yM9OoqK71ugxjPOGr8Ldnbz9Ewm9bfhNMvgp/u7v9YQu/CS5fhb+9cjPT2F3bQG19o9elGJNwgQ8/QKXt9DMB5Kvwd2TMD3agjwkmX4W//WN+55h+C78JIl+Fv70ObPmt228CKNDhzw47Z/bZlt8EUaDDnxJKIisj1cJvAslX4W/vDj+wz/pNcPkq/O3d4Qd2fL8JLl+FvyNyM9PszD4TSIEP/7FZ3dlWtc+O8jOBE/jwn9QnTJPChoo9XpdiTEL5Kvwd2eE3uHcmACXl1fEqy5guyVfh78gOv4E5GYSShJLtNXGszJiux1fh74jU5CTysrvblt8ETuDDD07X37b8Jmgs/Dg7/Tbt2MP+Btvjb4LDwg8M6pNJk8LGStvjb4LDwo+z5Qf42Lr+JkAs/Dh7/JME1m23nX4mOHwV/o58zg+QlhwiLzuD1dss/CY4fBX+jnzOHzH6xGzeWldph/mawPBV+I/E2KGfY199I2+UVHpdijEJYeF3jTohmx7pySwo/sTrUoxJCAu/KyWUxFdO6cPitdtpaGzyuhxj4s7C38yFp32OXXvrWbFxp9elGBN3Fv5mvnRSLplpyTywZB2q6nU5xsSVhb+Zbqkh7hx/Mm+v38GsglKvyzEmriz8LVx91nF8fmAWv3tpDR/bQT/Gxyz8LSQlCX+88gzSkpP42gNv8vd3N9tn/8aXxE9jWxG5BLhk0KBBN5SUlBzRssqra/mvmSt5c10lmWnJnHdSDkP69KD/Md3Izkile2qI9JQQKaEkkkNCkoCIIE4dTj0H6jqiUj5D6OQFdiGdva78LJQk9OvV7ZD3EZFCVc1vdZ6fwh+Rn5+vBQUFR7ycpiblrfWVzC3ayvKNOyn9dC8+XF3mKHVcVneW3X7BIe9zqPAnx6Uqn0hKEs4bnMt5g3MB2FfXSHl1LTv31LG3rpF9dY00NCkNTU2oQpPqgTcHxbnR2W8Wfn7z8fFTi4uM1NARPd7C3w7dUkMcn53B8dkZXpdizBGzHX7GBJSF35iAsvAbE1AWfmMCypcf9YlIBbC5jdk5QFc6ad/qaVtXqgWOznqOV9Xc1mb4MvyHIiIFbX3u6QWrp21dqRbwXz3W7TcmoCz8xgRUEMP/qNcFtGD1tK0r1QI+qydwY35jjCOIW35jDAEKv4iMFZGPRGSdiNzpwd8/VkSWiMgaEVklIre407NE5FURKXF/H5PgukIi8oGIvOi2B4rIcreemSKSmsBaeonIsyKy1l1Po71cPyLyY/d/VSwiM0QkPZHrR0SeEJFyESluNq3V9SGO+93X94ciMuJwyw9E+EUkBDwIjANOBa4WkVMTXEYD8BNVPQUYBdzo1nAnsFhVBwOL3XYi3QKsadb+A/BXt55PgesTWMt9wAJVPRk4063Lk/UjIv2Bm4F8VR0KhIBvktj1Mw0Y22JaW+tjHDDY/ZkETD3s0lXV9z/AaOCVZu3JwGSPa5oDjAE+Avq60/oCHyWwhgHuC+g/gBdxrj9SCSS3tt7iXEsPYCPufqhm0z1ZP0B/oBTIwjn79UXgokSvHyAPKD7c+gAeAa5u7X5t/QRiy0/0HxlR5k7zhIjkAcOB5UAfVd0G4P7uncBS7gVuByJfVJAN7FLVBredyPV0AlABPOkOQx4XkQw8Wj+qugX4E/BvYBtQBRTi3fqJaGt9tPs1HpTwt3ZxKE8+5hCRMDAbuFVVd3tRg1vHV4FyVS1sPrmVuyZqPSUDI4Cpqjoc2EPih0AHuGPpS4GBQD8gA6dr3VJX+bis3f+7oIS/DDi2WXsAsDXRRYhICk7wp6vqc+7k7SLS153fFyhPUDnnAF8TkU3AP3G6/vcCvUQkcpGXRK6nMqBMVZe77Wdx3gy8Wj9fATaqaoWq1gPPAV/Au/UT0db6aPdrPCjhfw8Y7O6pTcXZcTM3kQWIc1XPvwFrVPUvzWbNBb7t3v42zr6AuFPVyao6QFXzcNbHa6p6LbAEuNKDej4BSkVkiDvpy8BqPFo/ON39USLS3f3fRerxZP0009b6mAt8y93rPwqoigwP2pSInSdd4QcYD3wMrAd+7sHfPxenG/YhUOT+jMcZZy8GStzfWR7Udj7wonv7BGAFsA74PyAtgXUMAwrcdfQCcIyX6we4G1gLFAN/B9ISuX6AGTj7G+pxtuzXt7U+cLr9D7qv73/hfEpxyOXbEX7GBFRQuv3GmBYs/MYElIXfmICy8BsTUBZ+YwLKwm8SSkTOj5xBaLxl4TcmoCz8plUiMlFEVohIkYg84p73XyMifxaR90VksYjkuvcdJiLvuueRP9/sHPNBIrJIRFa6jznRXXy42Xn7090j6BCRKSKy2l3Onzx66oFh4TefISKnAFcB56jqMKARuBbn5Jb3VXUE8DrwK/chTwN3qOoZOEeXRaZPBx5U1TNxjouPHG46HLgV59oKJwDniEgWcDlwmruc38X3WRoLv2nNl4GRwHsiUuS2T8A59Xeme59ngHNFpCfQS1Vfd6c/BXxRRDKB/qr6PICq1qrqXvc+K1S1TFWbcA5zzgN2A7XA4yLydSByXxMnFn7TGgGeUtVh7s8QVf11K/c71LHhrZ1iGrG/2e1GnItjNABn45z1eBmwoJ01m3ay8JvWLAauFJHecOC6ccfjvF4iZ7RdA7ypqlXApyJynjv9OuB1da5VUCYil7nLSBOR7m39Qfc6Bz1VdT7OkGBYPJ6YiUo+/F1M0KjqahH5BbBQRJJwziq7EecCG6eJSCHOlW2uch/ybeBhN9wbgO+6068DHhGR37jL+MYh/mwmMEdE0nF6DT/u5KdlWrCz+kzMRKRGVcNe12E6h3X7jQko2/IbE1C25TcmoCz8xgSUhd+YgLLwGxNQFn5jAsrCb0xA/T/Ol3rZ2V9tqgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "L2 norm of w: 0.3548284\n"
     ]
    }
   ],
   "source": [
    "fit_and_plot_tf2(3, lr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 小结\n",
    "\n",
    "* 正则化通过为模型损失函数添加惩罚项使学出的模型参数值较小，是应对过拟合的常用手段。\n",
    "* 权重衰减等价于$L_2$范数正则化，通常会使学到的权重参数的元素较接近0。\n",
    "* 权重衰减可以通过优化器中的`weight_decay`超参数来指定。\n",
    "* 可以定义多个优化器实例对不同的模型参数使用不同的迭代方法。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
